Trading Strategies of a Champion Agent in a Multiagent Smart Grid Simulation Platform

Abstract

Local producers and storage units will play a major role in the future electricity grid along with the challenge of sustainability. For this reason, smart grid simulations are needed to forecast the challenges of two-way information and energy flow. The Power Trading Agent Competition (Power TAC) provides an open source simulation platform to enable and verify various smart grid studies from the perspective of sustainability. Besides, an annual competition is hosted in which autonomous agents trade in energy markets and make profits. AgentUDE won the Power TAC 2014 Final as a broker utilizing an adaptive agent. This paper details the trading strategies of AgentUDE and analyzes the tournament.

1 Introduction

Smart grid has been turning into an exciting area for researchers and business entities as new power players, such as electric vehicles and power to gas units, involve in the electricity grid that make it possible to store electricity in a distributed way. On the other side, some of governments started to declare their energy transition policies such as Energiewende in Germany: Within Energiewende, 17 nuclear power plants will be permanently shut down by the end of 2022 [9]. Meanwhile, fossil fuel based electricity production is likely to be replaced with renewable energy production, which has a fitful energy production volume [5]. In the light of this energy transition policies, information and energy flow between these energy actors have to be simulated within a realistic smart grid simulation to identify future challenges and propose solutions. Power TAC provides an open source, smart grid simulation with the aim of addressing a solution to this challenge through making autonomous brokers trade in a smart grid environment. Alongside, it simulates the typical energy markets, such as wholesale, retail and balancing markets (details are explained in Sect. 3) [1].

This paper addresses the wholesale market, retail market and balancing activities of AgentUDE. In particular, the main focus of this paper is the aggressive tariff strategy and contributions of tariff fees where analyses showed that AgentUDE gained the serious portion of its cash balance through early withdrawal penalty (EWP) and bonus payment (BP). AgentUDE won the Power TAC 2014 Final games as the newest participant among seven brokers by earning the most profit. Behind AgentUDE, cwiBroker and CrocodileAgent took the second and third places, respectively [7, 8].

The structure of the paper is as follows. Specifics of Power TAC 2014 Final games are introduced in Sect. 2. Afterwards, related work is given in Sect. 3. Section 4 is dedicated to AgentUDE that details the retail, wholesale and balancing activities. Future work is identified in Sect. 5. Finally, the paper is concluded in Sect. 6 with an outlook to Power TAC 2015 Final games.

2 Power Trading Agent Competition and 2014 Final Games

The Power Trading Agent Competition (Power TAC) is an open source smart grid simulation platform which consists of a wholesale market, a tariff market, a distribution utility and a number of costumer and producer models. Autonomous brokers are also allowed to trade remotely in these markets. The wholesale market is a typical day-ahead market where the large generator companies, renewable production farms and brokers place their bids and asks for the future time slots. The retail market allows brokers to build their customer portfolio by means of offering multiple tariffs to local producers and consumers. In between retail and wholesale markets, the distribution utility keeps track of supply and demand, and charges brokers for their energy imbalances. Customers are simulated as independent consumer and producer models for goods including electric vehicles, households, storage units and solar panels. The interaction between customers and brokers takes place in the retail market through tariff subscriptions. Figure 1 illustrates the schematic landscape of the Power TAC environment.

Fig. 1.

Components of the Power TAC as well as autonomous brokers. Each component represents a different module.

Brokers represent the business entities in the simulation platform. They offer tariffs through the tariff market, and their goal is to increase the number of their subscribed customers. With this aim, they have to trade in the wholesale market, in order to match their total supply and demand for a particular future hour. Enabled future time slots are declared by the wholesale market in advance. At the beginning, each game starts under monopoly conditions, with a built-in broker called default broker. In order to create initial market environment, this broker trade in the markets before the login of autonomous brokers. This interval is called bootstrap period. Afterwards, the competing brokers are allowed to join in the game. Note that timing in the simulation platform is not continuous. Rather, the simulation time progresses in discrete time slots. Each time slot is equal to five seconds in the real world, and one hour in the simulation world [1].

In the Power TAC 2014 Final games, 72 games were played, and 7 brokers competed. Out of these, 16 games were 8-sized, 35 games were 5-sized and 21 games were 3-sized. As stated already, Power TAC has a built-in default broker which is always included in all game sizes. All the statistics and data that are included in the paper are collected through the extraction of game logs which are produced after each game by Power TAC server.

Table 1 shows the official results of Power TAC 2014 Final games [10]. In total, 7 brokers competed in the tournament. Unfortunately, TacTex is not included in the official result since the TacTex team decided to withdraw its broker from the tournament due to some connectivity problems. At a first glance, it can be clearly seen that AgentUDE and cwiBroker dominated the games. AgentUDE took the first place in game size 5 and game size 8, and third place in game size 3.

Table 1. Official results of Power TAC 2014 Final. Values represent the normalized total profits of brokers. Final ranking is formulized through summing all game sizes [10].

Figure 2 illustrates the wholesale market trading patterns of the brokers, in which generator companies and other wholesale actors are excluded. Here, a negative price indicates broker payment for a certain amount of bought energy. Similarly, a positive price refers to a received payment for a certain amount of sold energy. Colors indicate the time proximity. Red color represents the far future in the simulation time, at which contracted energy will be delivered (up to 24 h). Similarly, blue color indicates the near future for a sooner delivery. Although there are minor differences between game sizes, the main characteristics of the market can be identified easily: As seen on the graph above, the cheapest energy is usually available at the last enabled time slot. After the last enabled time slot, the most expensive interval starts: It means that wholesale energy is sold immediately whenever it is available.

Fig. 2.

Cleared wholesale market prices of brokers.

The area to the right of the origin shows the selling activities of the brokers. It is not as active as the left side, since the priority of a broker is to match demand and supply. Very few brokers, such as Maxon, preferred to make brokerage in the market. Individual performances are detailed in Sect. 4.

Figure 3 presents the price trends of the brokers in the retail market. Apparently, brokers have their own individual price regimes, depending on their customer portfolio. What can be clearly seen here is that the hard competition takes place around 0.06 ₵/kWh. Further analysis can be found in the next section.

Fig. 3.

Tariff minimum values of competing brokers.

3 Related Work

Power TAC publications can be classified into two groups: Reviews and broker descriptions. One of the broker papers has been published by the TacTex team to introduce their broker which won the Power TAC 2013 Final games. As explained in this paper [3], TacTex uses Markov Decision Processes to minimize the energy costs in the wholesale market. Besides, it optimizes the future demands, prices and predicted energy costs in order to pick an appropriate tariff among pre-created, fixed-rate candidate tariffs. Another broker publication by the AstonTAC team focuses on wholesale market trading, using Markov Decision Processes for price optimization and Non-Homogeneous Hidden Markov Models for future predictions [4]. The last broker paper is by the cwiBroker team, which was very successful in 2013 and 2014 Final games: They took the second places in both tournaments, utilizing a trading technique that uses the equilibrium in continuous markets [8]. The most comprehensive review paper to date has been published by Jurica Babic and Vedran Podobnik for the Power TAC 2014 Final games. In this paper, brokers are compared based on the pre-defined key performance indicators (KPI). Besides, retail and wholesale market activities, including market shares and proximities of the future time slots, are discussed in detail [7].

Compared to the broker approaches above, AgentUDE implements an adaptive method in the wholesale market. On the retailer side, it uses an empirical strategy, which is inspired by the German electricity retail market. Within this strategy, tariff fees are speculated along with aggressive tariff publication. These methods and strategies are detailed in Sect. 4.

4 AgentUDE at a Glance

The broker abilities of AgentUDE can be divided into three groups: wholesale, retail and balancing market activities. As shown in Fig. 4 below, AgentUDE evaluates and completes its basic facilities with a time slot. In the following, we address these activitiy groups.

Fig. 4.

AgentUDE activities in a time slot.

The wholesale trading module of AgentUDE uses an adaptive method which tracks the past market data. Thanks to this method, the broker is able to catch the market trends regardless of weather conditions. However, statistics of the competition showed that wholesale market costs of brokers are very close to each other (See Table 2). Therefore retail activities are detailed more due to the diversity in tariff publication policies of the brokers. AgentUDE uses an aggressive tariff strategy by means of offering the cheapest tariff and speculating on tariff parameters (contract length, EWP and BP). There are two main goals in the retail strategy: To provoke other brokers to lower their tariffs and incentivate customers to change their tariffs. Eventually, this liquidity triggers the tariff penalties and results in profit. The results of this strategy are given in the next sub-sections.

Table 2. Wholesale market averages of the brokers.

Before turning to the wholesale market activities, an indicator used by AgentUDE has to be introduced here. It indicates the profit achievement acceleration of the broker, where a higher value means better profit performance. The idea behind it is to improve decisions in tariff creations and wholesale market activities. The following formula (1) evaluates the rhythm value at time slot t:

$$ R_{t} = R_{t - 1} + \omega * \left( {\frac{{\left( {c_{t} - \sum\nolimits_{n = 0}^{5} {\left( {\frac{{c_{t} - n}}{5}} \right)} } \right)}}{{c_{t} }}} \right) $$

(1)

Where R is the rhythm at given time slot, C is cash balance and ω is weight which is set experimentally. The formula above returns a value based on the cash positions. This rhythm is smoothed with a weight value to avoid bounces. The main impact of this parameter on the tariff publication cycle and profit margin given in Formula 3.

4.1 Wholesale Market Activities

Wholesale trading is a vital issue for all brokers to minimize their imbalanced energy. Additionally, brokers are challenged to buy the cheapest possible energy in order to be flexible against their competitors. In the end, customers would like to subscribe to the cheapest tariff available from their profitability perspective.

Price prediction takes place in two steps: The base price is a predicted utilizing past data. Afterwards, the final price is differentiated using the base price. Following formula (2) returns the base price at current time slot t for a future delivery at time slot T.

$$ B_{t,T} = \left( {B_{t - 1,T} + C_{t - 1,T} } \right) * \left( {1 - \omega } \right) + \left( {F_{t,T} + max_{t} C(t,T) - min_{t} C(t,T) + R\left( {t,T} \right)} \right) * \omega $$

(2)

Where B is a base price for the given current time slot t and future time slot T. C stands for the market equilibrium price for t and T. R is risk function that contributes to the price depending on time slot proximity. F indicates constants such as market mean and averages. The weight, ω is updated using the rhythm value which is given in Formula 1.

Figure 5 illustrates the cleared bids and asks of AgentUDE. Overall, bidding density of AgentUDE is narrowed between 15 and 25 EUR/MWh. Consequently, the average buying price is realized at 22.7 EUR/MWh and selling price at 28.9 EUR/MWh (See Table 2). Surely, these cost prices make sense with imbalance activities. The cost can be easily decreased with a stingy bidding policy. However, this would eventually lead to poor imbalance performance.

Fig. 5.

Cleared bids and asks of AgentUDE.

Figure 6 illustrates the prediction performance of AgentUDE in different game sizes. In 8-sized games, the success rate is higher than other game sizes since the market is more stable due to the large number of participants. Before fifth future time slot, selling prices are always less than buying prices. Therefore, this area is regarded as a risk area due to approaching delivery time and brokers may submit extraordinary prices in order to avoid imbalance penalties. These panic orders can clearly be seen in Fig. 2 as blue colored prices that are close to 100 EUR/MWh.

Fig. 6.

Average cleared wholesale prices and trading performance of AgentUDE.

Table 2 lists the wholesale bidding and selling costs of brokers. P _bids and P _asks indicate the average bidding and asking prices. AgentUDE has a market cost around 22.7 EUR/MWh and asking performance of 28.9 EUR/MWh, where the consumption share is 22.9 % of the total energy distribution. In this landscape, AgentUDE is the second best broker after cwiBroker in terms of the lowest market cost. However, these values are very close to each other and do not provide a serious contribution to the overall profits of the brokers. Instead, we take a closer look at retail activities in the next section, which makes AgentUDE stand out against the competition among other brokers.

4.2 Retail Market Activities

AgentUDE applied a new strategy on the retail side, which is not used by other brokers: Publishing aggressive tariffs with the lowest tariff values and customer binding tariff fees such as EWP and BP. Over the course of the competition, this strategy provoked other brokers to publish cheaper tariffs, which in turn triggered the EWP’s of AgentUDE tariffs. As a whole, AgentUDE forced its customers to change their tariffs. In the end, this strategy contributed about 20 % to overall cash balance (See Fig. 7).

Fig. 7.

Total cash position and cumulative sum of EWP and BP.

Table 3 shows the tariff statistics of the brokers. N _tariffs is the total published tariffs. Frequency indicates the publication cycle in terms of time slot. M _cons is the mean price of consumption tariffs. Similarly, M _prod is the mean of production tariffs. S _cons is the average price of energy that is sold to customers. Likewise, S _prod refers to the price for bought energy. Finally, E_cons and E_prod are energy consumption and production shares of brokers, respectively.

Table 3. Tariff activities of the brokers in Power TAC 2014 Final.

As seen in Table 3, AgentUDE published most of the tariffs having a publication cycle of 27. After AgentUDE, Mertacor and TacTex have most of the tariffs. On the other hand, only AgentUDE, CrocodileAgent and the default broker published production tariffs. However, the production tariff policy of AgentUDE is restricted with a simple rule. Only if the sum of minimum production tariff value and distribution fee is less than wholesale market cost, production tariffs are published. Otherwise, local producers are ignored.

All the games start with a number of uncertainties such as market status (production and consumption capacities) and the number of competitors. First of all, broker agents are not aware of their competitors’ trading strategies. Therefore, initial tariffs have to be set carefully. Following piece of code states the initial publications of AgentUDE.

Here, MM and DF are market mean price and distribution fee, respectively. These parameters are announced at the beginning of each game. Integer value of tariff number, tNum is included to “publishTariff” method in order to create different tariff variations. PM is profit margin, which is set heuristically. BC represents a function which takes the number of brokers into account: In case of high participation, tariff value, abbreviated as tVal, is decreased. According to the Power TAC specification [1], only the first five tariffs of each power type are visible to customers. Therefore five tariffs are initially published for the maximum exposure. Early withdrawal penalty is formulated as a function of EWP based on the number of brokers. Due to Inertia parameter described as I _a  = I * (1-2 ⁻ⁿ ) and valued between 0 and 1, customers are highly sensitive to the new tariffs at the beginning of the games. Therefore, EWP’s are extremely useful fees to bind customers to the tariff. Eventually, customers’ loyalty increases due to the Inertia parameter and they usually continue to stay within the tariff even if the tariff is not the cheapest one. As a part of retailer strategy, AgentUDE always set EWP if the tariff value, to be published, is the cheapest one in the market.

The simplified algorithm above describes the process how AgentUDE publishes new tariffs. Concisely, this method publishes two kind of consumer tariffs. If the offered price is the cheapest one among other tariffs, then it is published with a EWP fee. Otherwise, EWP is not set and tariff value is adjusted considering the market cost.

AgentUDE employs a number of controllers during the tariff publication process. One of these controllers is the “monopolyTest” method. This method is triggered if a price gap appears between AgentUDE and its closest competitor. Another controller is the “revokeUselessTariffs” method, which removes harmful tariffs. It is quite possible in a game that wholesale clearing prices increase due to high demand and weather conditions. In this case, some of the older tariffs might be outdated and harmful in terms of profitability. This method simply removes such tariffs.

Market cost is calculated by the “getCost” method and takes cleared wholesale market prices and distribution fee into account. Formula (3) shows the definition of the value computed by the method:

$$ getCost_{t} = \frac{{(\sum\nolimits_{n = 0}^{60} {\sum\nolimits_{m = 1}^{24} {p_{m,n} )} } }}{{(\sum\nolimits_{n = 0}^{60} {\sum\nolimits_{m = 1}^{24} {E_{m,n} )} } }} + DF + PM $$

(3)

Where DF and PM represent the distribution fee and profit margin, respectively. P and E refers to total money and energy transactions and the formula above runs up to the most recent 60 h and 24 enabled future auctions of the wholesale market. Consequently, the fraction yields an average cost price by means of dividing the total payment to the total energy. Another controller is the “getCompetitorsMinimum” method, which scans the tariff repository and identifies the competitors’ minimum tariff with a small margin. Other controllers are “getCriticalRate” and “getSubscriptionRate”; they represent the goal and current situation, respectively. The critical rate is the minimum percentage of total customers that AgentUDE has to reach; the subscription rate refers to the percentage of currently subscribed customers.

The subscription rate and critical rate shape EWP fees. The number of subscribed customers is proportional to EWP fees, based on the Formula (4). All together, the calculation of early withdrawal penalty fee can be formulated as follows:

$$ EWP_{t} = (getCriticalRate() * T) * \varphi - S * \omega $$

(4)

Where T is the number of total customers in the competition environment, and S is the number of subscribed customers. The weights, φ and ω differ based on the power type.

Figure 7 shows the overall cash balance and collected money from tariff fees as a result of the strategy. In the same figure, red area shows the cumulative sum of tariff fees which is the approximately 25 % of the overall cash position. This rate increases in 8-sized games due to stiff competition. In other words, high number of tariffs means higher liquidity in terms of customer subscriptions and withdrawals (see Fig. 9).

Figure 8 compares the tariff fee performances of all the brokers. Surprisingly, only AgentUDE and TacTex benefitted from tariff fees. Here, maximum profit achieved from 8-sized games.

Fig. 8.

Average cumulative profits that are collected from EWP and BP.

In order to earn more profit from this strategy, some requirements have to be met: Active customers and a stiff competition. First, customers have to see some profitable tariffs on the desk before leaving their current retailer. If not, customers tend to ignore the available tariffs and stick to their tariff. In this case, the strategy offered by AgentUDE does not work well. Second, a broker has to offer competitive tariffs, so that customers can see them and change their tariffs if it is really profitable for them. To illustrate this analysis, competitive and non-competitive brokers are tested in 3-sized games below.

Figure 9 draws the tariff fee earnings of AgentUDE as a result of 3-sized games between AgentUDE and the respected broker. Apparently, TacTex, CrocodileAgent and cwiBroker provided the most profit to AgentUDE while Mertacor, Maxon and coldbroker did less. In the same fashion, this symbiotic relationship is proportional to the official results given in previous sections. Another result is that TacTex, cwiBroker and AgentUDE offer the most profitable tariffs to the customers and convince them to change their tariffs.

Fig. 9.

Cumulative tariff fee earnings of AgentUDE that are collected through 3-sized games.

4.3 Balancing Activities

Brokers have to meet their demand and supply. If not, they may lose the largest portion of their profits by paying huge imbalance fees. The most challenging issue is to predict future consumptions. AgentUDE uses the consumption data of customers to make predictions. However, this method does not always give the best result due to changing conditions, including weather. Balancing market sends signals to brokers regarding their imbalance status. Accordingly, needed energy is calculated as the sum of predicted consumption and imbalance signal. The final amount of needed energy is smoothed and submitted to the market.

Figure 10 illustrates the cumulative imbalance volumes. In this figure, negative and positive volumes are regarded as absolute values and they are summed regardless of their signs. Apparently, 3-sized games give the best result for AgentUDE. Since the figure illustrates the volumes, increasing number of participants makes it difficult for AgentUDE to adjust its imbalance due to changing demand. Besides, customers have more tariff options in game size 8 in comparison to game size 3. Therefore, withdrawal or sign-up activities of customers eventually result in last-minute imbalances.

Fig. 10.

Cumulative volume of negative and positive imbalances.

Figure 11 shows the average imbalances where negative and positive values are summed. AgentUDE draws a flat line due to wave-style imbalance activity. In other words, positive and negative values are almost same. Figure 12 illustrates the overall imbalance payments from brokers to the distribution utility. AgentUDE is the second best broker in terms of paying the least money to the distribution utility. However this payment only consists of imbalance penalties since the total imbalance energy is close to the zero line.

Fig. 11.

Cumulative sum of positive and negative imbalances (all game sizes).

Fig. 12.

Total imbalance payments from brokers to distribution utility (all game sizes).

For a typical negative imbalance, brokers have to pay the sum of penalty fee and price of imbalanced energy. TacTex and cwiBroker paid 100 k EUR for their 1700 MWh and 1450 MWh imbalanced energy, respectively. If a comparison is needed at 70 k EUR, where the imbalanced energy of AgentUDE is almost zero, TacTex and cwiBroker paid 70 k EUR plus 17.6 EUR/MWh and 20.6 EUR/MWh, respectively for their negative imbalance. With respect to the wholesale market costs shown in Table 2, TacTex and cwiBroker had a good deal on the balancing market over AgentUDE.

5 Future Work

AgentUDE seems to be a promising broker. However, there are issues to be improved upon. Following points are the most important topics that are expected to be solved for the upcoming Power TAC 2015 games.

One of the most important issues is efficiency in wholesale trading. AgentUDE still loses sizeable amounts of money through its relatively inefficient bids and asks. Therefore, price predictions in the wholesale market ought to take weather forecasts into consideration, in order to catch such future trends.

A second improvement regards the utilization of unused power figures. In the Power TAC environment, there are many new generation power actors, such as storage units, controllable customers. However, most of the brokers do not use them. No doubt, utilizing these components improves the efficiency of the broker.

Another issue is capability of surviving in a longer game period: As asked for the future competitions, AgentUDE has to be compatible for longer games.

6 Conclusion

This paper covers the basic strategies of AgentUDE and results of the competition from the business perspective, as a winning agent in Power TAC 2014 Final. However, success is a relative term, especially on such a dynamic and progressive platform. The participating teams get stronger year by year, and change their strategies. As a result, the competitiveness of the game is raised aggressively. For this reason, comparisons are valid only for the specific releases of participating brokers.

As has been noted in the wholesale market section, the gaps between the market performances of brokers are very close to each other. It is clearly seen that all the brokers have a decent market performance based on their customer profiles and risk levels. What placed AgentUDE one step ahead are its competitive and aggressive tariff strategies. In addition, the results showed that AgentUDE earned the serious portion of its profit through tariff fee speculation. This strategy was never used before by another broker and turned AgentUDE into a more competitive and flexible competitor. Lastly, the Power TAC 2014 Final showed that it has an enormous benchmark potential for smart grid studies. Therefore, we kindly invite new teams to take part in this competition.