Financialization of the Commodities Markets: A Non-technical Introduction

Abstract

The goal of the first part of this chapter is threefold: (a) to introduce the term structure of forward/futures commodity prices, the contango/backwardation duality and the notion of rolling yield as it pertains to trading through commodity indexes; (b) to use principal component analysis and the computation of equity and commodity “betas” to provide empirical evidence of the dramatic changes which occurred in the mid-2000s; (c) finally, to review the major arguments which have been put forth in the debate over the financialization of these markets. While conspicuously absent from some of the English language dictionaries, the word financialization has been widely used to describe the increasing role of institutional investors in the commodity markets. Using econometric data analyses for the purpose of illustration, we concentrate on futures price data from the post-2004 period during which the commodity markets experienced a significant influx of new financial investors. As far as we know, mathematical models attempting to reproduce or illustrate (let alone explain) the empirical observations at the core of the debate are few and far between. As a result, our approach remains mostly descriptive of the data which have been used to back up the claims of the various sides of the argument. The originality of our contribution, if any, is the discussion of a new generation of roll yield maximizing commodity indexes, the empirical analysis of the term structure of open interest, and the possible connections between the two.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

The main goal of this chapter is to document the dramatic changes in commodity prices during the post-2004 period, when commodity markets experienced a large influx of new money, especially from institutional investors. For the sake of completeness, we review some of the idiosyncrasies of these markets as well as the main data analysis techniques used to study the term structure of forward prices, our objective being to focus on the changes which occurred over the last decade. We rely on economic studies to explain why, according to the restricted form of the financialization hypothesis, they produced changes in correlations, and rises in open interest and trading volume. Whether this increase in open interest and volume is due to index investing or herding behavior is still unclear. We demonstrate the increase in trading volume and open interest throughout the period, and we analyze the term structure of commodity open interest. We use Principal Component Analysis to demonstrate the shift of open interest down the curve. This increase in open interest along longer maturities coincides with the appearance of a new generation of commodity indexes optimizing the roll yield. While the compositions of the portfolio covered by these indexes is pretty much the same as the compositions of the traditional indexes, the spectrum of contract maturities they comprise is different because of the special nature of the rolling algorithms. While we cannot prove causality between the appearance of these new indexes and the sliding down the curve of the open interest, we highlight their simultaneity as food for thought.

First, we start by defining the meaning we shall give to the term financialization which according to the New Oxford American Dictionary means the process by which financial institutions, markets, etc., increase in size and influence. In this chapter, we talk about the financialization of commodities to mean the increased role of financial markets in the operation of the commodities markets. For the purpose of this chapter, we restrict the scope of this definition and use the terminology financialization hypothesis to mean that the sharp increase in volatility and the price hikes observed in the commodity markets between 2004 and 2008 are due to the overwhelming influence of large institutional investors using indexes to gain exposure to commodities, and not to an imbalance in supply and demand for physical commodities due to the growth in emerging economies such as China, India and Brazil.

While there is no clear rhyme or reason for the timing of the emergence of this financialization, it is widely accepted to be associated for the most part, with the appearance of a new class of large investors who chose to take positions on commodities as a group, in order to capture profits considered to be unattainable from investments in more traditional assets. Treating commodities at the same level as stocks, bonds, real estate, etc. they promoted commodities to the rank of a new asset class.

This spectacular increase in investment in the commodity markets by investors whose primary business or financial interests were not directly dependent upon changes in the prices of the physical commodities was treated as pure speculation, and has been the source of heated discussions among economists, policy makers as well as in the media. Case in point, the 2008 bubble in the prices of a wide range of commodities as shown in Fig. 1 with the plot of the evolution of a global commodity index representative of the spot prices of a large group of commodities. Details on the construction of the index plotted in Fig. 1 will be given in Sect. 3. As we are about to explain, this bubble has caught the attention of policy makers and focus their investigations on the roles of the various groups of financial investors in the commodity markets.

Fig. 1

Time series plot of the GSCI daily spot index

The emergence of specialized indexes and the growth in popularity of long-only index-fund investing are some of the remarkable differences between commodities and other asset classes. According to Barclays’ internal reports, in 2006–2007, index fund investment increased from 90 billion to 200 billion USD. Simultaneously, commodity prices increased 71 % as measured by the Commodity Research Bureau. At the peak of the price bubble in 2008, commodity fund investors, including ETFs and hedge funds like Soros Fund Management, controlled a record 4.51 billion bushels of corn, wheat and soybeans through the futures markets of Chicago Board of Trade, equal to half the amount held in U.S. silos on March 1, 2008. In his testimony before the U.S. Senate Commerce Committee, George Soros stated that commodity investment, as a new venue for institutional investors, had become ‘the elephant in the room’ and as a result, investment in these assets might exaggerate price rises. After the price collapse which occurred between June 2008 and early 2009, many pundits referred to this boom and bust as a bubble as futures prices far exceeded fundamental values. The large scale speculative buying by index funds was held as culprit. A number of studies on financial markets have suggested that herd formation among large institutional investors may have destabilized market prices and created excess volatility (see for example Dennis and Strickland [10], and Gabaix et al. [12]). From these studies, one can argue that herd behavior in the commodity markets, as driven by financial investors moving funds in and out of commodities, was a contributing factor behind the booms and busts observed in a wide range of commodities.

On the other end, some economists (including Nobel Prize winner P. Krugman [22], Irwin and Sanders [16], Hamilton [13] and Kilian [19]) remained skeptic about the bubble theory. They argue that commodity price cycles are driven by fundamental factors like supply and demand, and that the temporary imbalances observed in 2008 are due to the spectacular growth in emerging economies. Adding support to this view, Buyuksahin and Harris [4] examine the trading positions of various types of traders in the crude oil market, and find little or no evidence that financial investors’ position changes caused price changes in the oil market.

This did not stop commodity index investing from being under attack. Increased participation in futures markets by non-traditional investors was deemed disruptive and blamed for the 2007–2008 “Food Crisis” that is at the origin of the famous “Casino of Hunger: How Wall Street Speculators Fueled the Global Food Crisis” [11]. See also [3]. A report from the U.S. Senate Permanent Subcommittee on Investigation “… finds that there is significant and persuasive evidence to conclude that these commodity index traders, in the aggregate, were one of the major causes of unwarranted increases in the price of wheat futures contracts relative to the price of wheat in the cash market…..”. To add insult to injury, a group of 48 agriculture ministers meeting in Berlin said that they were “… concerned that excessive price volatility and speculation on international agricultural markets might constitute a threat to food security….”, according to a joint statement handed out to reporters on January 22, 2011.

Broadly speaking, the financialization of commodities should refer to the increased leverage and the exponential growth of financially settled contracts dwarfing their physically settled counterparts. More recently, this term has also been used to refer to the significant impact of index trading on commodity prices, and even more narrowly speaking, to the increased correlations between the commodities included in the same index, and also between equity returns and commodity index returns. This last fact is illustrated in Fig. 2 which shows the time evolution, as given by a Kalman filter, of the time-dependent “beta” of the least squares linear regression of the Goldman Sachs Commodity Index Total Return against the returns of the S&P 500 index. Instantaneous “betas” are typically computed using Kalman linear filters as estimates of the slope of a local linear regression whose domain varies with time. See for example section 7.5.2 entitled Linear Models with Time Varying Coefficients of the textbook [6] for details. The standard commodity indexes are reviewed in Sect. 3, and a new generation of roll yield optimizing indexes is introduced in Sect. 5.

Fig. 2

Instantaneous dependence (β) of the daily GSCI-TR returns upon the corresponding S&P 500 returns

It is an empirical fact that return correlations are no longer what they used to be, and it is now commonly accepted that correlation in price changes for commodities included in the same index tightened before 2007. Tang and Xiong [32] argue that commodity index trading is responsible for this correlation tightening. See also [9], the works of Irwin and Sanders [16, 29], and especially [28, 30] for the impact of index trading on the agricultural commodities. This restrictive form of the financialization hypothesis is discussed in Sect. 4.

Note that it is likely that this correlation tightening is a scale dependent phenomenon. Indeed, it seems that high frequency traders do not see (and hence ignore) these correlation increases.

Commodity contract valuation is best understood by equilibrium arguments based on supply and demand for the physical commodities. In [7], we advocate structural models for the pricing of commodities and commodity derivatives. However, one of the main contention of the financialization of commodities is that the pricing models based on matching supply and demand are impaired by the overwhelming sizes of trades by institutional investors which increase price volatility and drive prices away from the levels predicted by fundamental supply and demand relationships. As a result, commodity price dynamics no longer merely reflect changes in fundamentals.

These conflicting views are yet to be reconciled, and investor behavior in the commodity markets needs to be further investigated, especially for the role it plays (if any) in the excessive price movements observed so frequently. The dramatic increase of commodity trading volume (often referred to as the financialization of commodity trading) occurred essentially at the same time as demand for physical commodities from emerging economies increased rapidly. The simultaneity of these two contributing factors make it extremely difficult to parse out their relative contributions to the increased volatility of the markets, and disentangle their respective price impacts.

We close this introduction with a short summary of the contents of the chapter. Section 2 offers a crash course focusing on the idiosyncrasies of the commodity markets, while Sect. 3 takes an historical perspective to introduce the traditional commodity indexes. The influx of institutional investors in the commodity markets and the changes they are responsible for are documented in Sect. 4 where a short review of the publications on the financialization debate is provided. Section 5 introduces the new generation of commodity indexes designed in most part for the purpose of maximizing the roll yield, and Sect. 6 documents the changes in the term structure of open interest as food for thought as a possible impact of the growing popularity of these new indexes.

2 Generalities on the Commodity Markets

In order to set the stage for our discussion of the financialization hypothesis, we review some of the basic idiosyncrasies of the commodity markets, focusing on those relevant to the debate. As already emphasized in the introduction, the role of institutional investors is paramount to the discussion of financialization of the commodity markets. The large number of commodities, the large number of venues on which these commodities are traded, added to the great variety of contract maturities, physical commodity grades and delivery locations offer a wide range of opportunities for hedgers and speculators. As a result, liquidly traded contracts represent a rather small part of the commodity world. However, they are most likely to be included in the commodity indexes, and traded for purely speculative purposes. Consequently, they will be our favorite targets when we look for illustrations of some of the claims used in the financialization debate.

2.1 The Markets and the Trades

Because of the physical nature of the interest underlying the contracts, commodity prices are determined by equilibrium arguments which involve matching supply and demand for the physical commodity itself. On the supply side, estimating and predicting inventories and quantifying the costs of storage and delivery are important factors which need to be taken into account. This is not always easy in the context of standard valuation methods which are mostly based on traditional finance theory (think for example of NPV which attempts to compute the present value of the flow of future dividends).

Whether they were called spot markets (when they involved the immediate delivery of the physical commodity), or forward markets (when delivery is scheduled at a later date), commodity markets started as physical markets. Trading volume exploded with the appearance of financially settled contracts. While forward contracts are settled Over the Counter (OTC), and as such, carry the risk that the counterparty may default and not meet the terms of the contract, most of the financially settled contracts are exchange-traded futures for which the exchange acts as clearing house controlling default risk by a system of margin calls, and attracting speculators to provide liquidity to the markets. While trading in physically and financially settled contracts were traditionally the two ways an investor could gain exposure to commodities, the creation of indexes and the increasing popularity of index tracking Exchange Traded Funds (ETFs) have offered a new way to gain exposure to commodities.

In the early 2000s, investing in commodities was promoted as a fool-proof portfolio diversification tool. After all, these financial interests were believed to be uncorrelated or negatively correlated with stocks. Case in point, the prospectus of the S&P GSCI (see the section on commodity indexes for details on the definition and the properties of this index) claims “… and provides diversification with low correlations to other classes”.

The exponential growth of this new form of investment in commodities which took place over the last decade may have been a self-defeating prophecy as recent econometric studies have shown that this form of index trading has created new correlations between commodities and stocks, and between the commodities included in the same index [32]. Pushing the argument even further, one could posit that the influence of investors has overturned Keynes’ theory of normal backwardation ¹ (see for example [17], or [18] for a more modern account, and [8] for a discussion focused on agricultural commodities), causing a recent predominance of forward curves in contango, thus further weakening the attractiveness of investing in these markets. We explain the duality contango/backwardation in Sect. 2.4 below.

One of the many convenient features of commodity trading is the specialization of the exchanges, leading to simple correspondences between commodities and locations where they are traded. In other words, a given commodity is traded on one or a small number of specialized exchanges. This is in sharp contrast with the equity markets for which a given stock can be traded on many platforms, leading to subtle optimization problems as the choice of a particular venue for a trade can affect the profits or losses on the trade.

The following table gives a few examples of some of these exchanges in the US and in Europe.

Table 1

2.2 Trading Commodities

Traditionally, the investment portfolios of large institutional investors (e.g. pension funds and endowment funds) included only stocks, bonds, and cash. The primary advantage of including commodities is that commodity returns are expected to be relatively uncorrelated with the returns of traditional asset classes. The absence of correlation is attributable in part to inflation. In fact, holding commodity futures is often considered to be an inflation hedge. Indeed, during periods of rising inflation, traditional asset categories like stocks and bonds perform poorly. Commodities, on the other hand, generally perform well during these periods. Indeed, increased demand for goods and services, typical in periods of rising inflation, usually implies increased demand for the commodities used in the production of those goods and services.

There are several ways in which traditional investors used to gain exposure to commodities.

• 1. The old-fashioned way to invest in commodities is to actually purchase the physical commodity itself. However most investors are not ready or equipped to deal with issues of transportation, delivery, storage and perishability. This form of involvement in commodities was created for, and is essentially limited to, the hedgers who mitigate the financial risks associated with uncertainties in the production and delivery of commodities relevant to their businesses.

• 2. Another way to gain exposure to commodities is to invest in stocks in commodity intensive businesses: for example buying shares of Exxon or Shell as a way to invest in oil. Many exchange traded funds (ETFs) are tracking portfolios of stocks of companies with well defined commonalities. The portfolios of a large number of these ETFs comprise only energy companies, and as such, they call themselves commodity ETFs. They promote themselves as investment vehicles to gain exposure to commodities despite the fact that they are technically equity ETFs. However, this type of investment offers at best an indirect exposure as shares of natural resource companies are not perfectly correlated with commodity prices.

• 3. A more direct form of exposure to commodities is through straight investment in commodity futures and options. The exchanges offer transparency and integrity through clearing, and relatively small initial investments are needed to take large positions through leverage. However, this convenience comes at a serious price, as discovered by many rookies who ended up choking, unable to face the margin calls triggered by adverse moves of the values of the interests underlying the futures contracts. Also, purely speculative investments of this type may need to be structured with a careful rolling forward of the contracts approaching maturity in order to avoid having to take physical delivery of the commodity: trading wheat futures can be done from the comfort of an office set up in a basement, but taking physical delivery of one lot (i.e. 5, 000 bushels) of wheat requires a large backyard! Consequences of some of the simplest rolling strategies are discussed in Sect. 2.4 below.

We first discuss the idiosyncrasies of commodity prices, and postpone to a later section the presentation of the more recent (and most relevant to the focus of this chapter) form of exposure to commodities based on index investing and/or tracking.

2.3 Data Used for Illustration Purposes

We use a specific set of commodities for the purpose of illustration. We chose Crude Oil because of its overwhelming impact on the global economy, and Copper as an example of metal. Copper is widely accepted by economists as a representative commodity because historically, it has been a consistent predictor of the health of the global economy, presumably because it is an important input in a huge number of industrial processes. Figure 3 gives a time series plot of the values of the nearest maturity Copper futures contract (as close as we can get from a spot price!).

Fig. 3

Time series plot of the daily price of the nearest copper futures contract between January 2, 1990 and September 9, 2013. Source: Data Stream

We use Light Sweet Crude Oil futures price data from NYMEX (part of the CME group) provided by Data Stream. These prices serve as a key international benchmark. The contract sizes are for 1, 000 barrels and the prices are quoted in US dollars and cents per barrel. Prices are quoted for monthly contracts with times to maturity up to 6 years. Trading in the nearest maturity contract ceases on the third business day prior to the 25th calendar day of the month preceding the delivery month. Delivery is free-on-board (FOB) at any pipeline or storage facility in Cushing, Oklahoma.

When discussing Copper, we use forward data, also from CME COMEX, and also provided by Data Stream. The contract sizes are for 25,000 pounds, and the prices are quoted in US cent per pound. While more forward contract prices are listed, we shall only use the nearest 23 maturity months. Trading in a contract with a given delivery month (maturity) ends on the third last business day of the delivery month. Note that these contracts are also traded on the London Metal Exchange (LME) and the Shangai Futures Exchange.

During the period 1998 through 2007, the trading volume in exchange-traded commodity futures and futures options experienced a five-fold increase. As an example, Fig. 4 gives the time series plot of WTI Crude Oil daily open interest. This plot represents on each day, the total number of contracts, irrespective of their maturities, held by investors. Corresponding plots (see for example Fig. 10 for the case of Copper) for other commodities would show the same dramatic increase, attesting the significant influx of money in commodities.

Fig. 4

Time series plot of the daily (total) open interest in WTI crude oil between January 3, 1994 and November 22, 2011. Source: Data Stream

However, most institutional investors do not have the sophisticated trading operations necessary to manage a complex portfolio of futures contracts: commodity index funds and OTC commodity return swaps appeared as attractive solutions. Both forms of investment are transparent and passive, so no need to monitor the market to identify underpriced commodities or timing profit opportunities.

2.4 Contango, Backwardation and the Roll Yield

We now introduce more of the jargon of the term structure of forward and futures prices in the form of a definition for easier reference.

Definition 1.

We say that the market is in backwardation, or that the forward curve is backwarded, when futures prices are lower than the expected future values of the spot price.

Because the futures prices must converge toward the expected spot price when approaching maturity of the contract, futures prices are rising to get in line with the expected spot price. Typically backwardation occurs when the left most part of the curve is downward sloping.

Definition 2.

We say that the market is in contango, or that the forward curve is in contango, when futures prices are higher than the expected future values of the spot price.

Because the futures prices must converge toward the expected spot price when approaching maturity of the contract, futures prices are falling to get in line with the expected spot price. Typically, contango occurs when the left most part of the curve is upward sloping.

We close this subsection with a formal definition of the roll yield, and a simple example showing that this yield is positive (resp. negative) when the forward curve is in backwardation (resp. contango).

Definition 3.

The roll yield is the return (profit or loss) captured by a market participant liquidating a long position in a contract approaching maturity, and taking the same position in the new nearest maturity contract.

The typical example to keep in mind is the profit (in which case the roll yield is positive) gained in backwardation, by merely maintaining a long position in the nearest contract. Indeed, in the case of a backwarded forward curve we have p ₁ > p ₂ if we denote by p ₁ and p ₂ the prices of the forward/futures contracts with the shortest maturities T ₁ < T ₂ after the current time t. Consequently, maintaining a long position in the nearest contract is done by closing the current position (i.e. selling at the unit price p ₁ the contract with maturity T ₁ as t approaches T ₁), and opening the same long position in the nearest maturity contract (i.e. buying the same amount of contracts with maturity T ₂ at the price p ₂), locking a profit, just for rolling the position to the new nearest maturity! So taking a long position in a backwarded market guarantees a positive roll yield, and hence a profit, just for rolling the same position from one maturity to the next when contracts approach maturity.

Similarly, maintaining (rolling) a long position in a contango market leads to losses, and hence a negative roll yield as a result. A transition from a backwarded market to a market in contango is one of the common fears of passive commodity traders.

Systematic studies of the nature of the roll yield can be found in the academic literature. As an example, the interested reader may want to look at [27].

2.5 The Term Structure of Forward Prices

On any given day t, the term structure of forward prices is given by the prices of the futures contracts for a given set $T_{1},T_{2},\cdots \,,T_{n}$ of maturity tenors. While t changes from one day to the next, the tenors $T_{1},T_{2},\cdots \,,T_{n}$ remain the same as long as t < T ₁. While the actual values of the maturity dates T _i are crucial to understand seasonal commodities such as natural gas or most of the grains, they can be a hindrance for many statistical data analysis techniques which require some form of stationarity in time of the data. On any given trading day, say t, if price quotes $p_{1},p_{2},\cdots \,,p_{n}$ are available for maturity dates $T_{1},T_{2},\cdots \,,T_{n}$, the points

$$\displaystyle{(T_{1},p_{1}),(T_{2},p_{2}),\cdots \,,(T_{n},p_{n})}$$

in the plane offer a discrete sampling of an hypothetical forward curve $T\hookrightarrow f(t,T)$ which could be defined for T > t. One of the problems is that when time passes by and t becomes, t + 1, t + 2, …, the maturity dates T ₁, T ₂, … do not change, and eventually t gets too close to T ₁ and the contract with maturity T ₁ ceases to be traded, and the nearest contract available for trading becomes T ₂. To avoid this sudden change in the input data, it is often convenient to re-parameterize the term structure of forward prices by the time to maturity $\tau = T - t$ instead of the time of maturity T.

2.5.1 Data Pre-processing

Switching from the parameterization by time-of-maturity to time-to-maturity requires extrapolation/smoothing and resampling of the forward prices. Below, we describe the steps we took to produce the numerical illustrations given in this chapter. Other procedures have been proposed to solve this issue. The problem is especially delicate in illiquid markets with a small number of quoted forward prices, and in highly volatile markets like the electricity markets. For example, the reader is referred to Chapter 7 of [2] for a detailed discussion of the latter.

On any given trading day, say t, we replace the maturity times T ₁, T ₂, … after t by the corresponding times to maturity $\tau _{1} = T_{1} - t$, $\tau _{2} = T_{2} - t$, …, $\tau _{n} = T_{n} - t$, and we plot the price quotes $p_{1},p_{2},\cdots \,,p_{n}$ against these values of τ. In other words, we consider the points

$$\displaystyle{(\tau _{1},p_{1}),(\tau _{2},p_{2}),\cdots \,,(\tau _{n},p_{n})}$$

as discrete sample observations of a hypothetical forward curve $\tau \hookrightarrow \tilde{f}(t,\tau )$ which could be defined for τ > 0. The main advantage of this re-parameterization of the curve is that its domain of definition does not change with t, and it is thus easier to have meaningful comparisons between forward curves on different days. On each day t, this hypothetical forward curve $\tilde{f}(t,\,\cdot \,)$ is often called a continuous maturity forward curve. It can be estimated by regression. In all the examples considered in this chapter, we used a standard cubic spline regression to produce continuous maturity curves. Modelling the term structure of forward prices by parametric families of classical functions is very convenient. This approach was successfully implemented for the analysis of the term structure of interest rates, and central banks, regulators and fixed income desks of major banks have developed their own proprietary methods to do so. But from a practical point of view, handling functions of a continuous variable is not always easy, and it is natural to work with discrete subsamples

$$\displaystyle{\tilde{p}_{1} =\tilde{ f}(t,\tilde{\tau }_{1}),\tilde{p}_{2} =\tilde{ f}(t,\tilde{\tau }_{2}),\cdots \cdots \,,\tilde{p}_{1} =\tilde{ f}(t,\tilde{\tau }_{m}),}$$

for a fixed set $\tilde{\tau }_{1}$, $\tilde{\tau }_{2}$, … which will not change from day to day. The choice of these fixed values of the time to maturity often starts with values like 1mo, 2mo, …, but these values do not have to be regularly spaced, and they do not have to be in the same number as the number n of original observations. The discrete forward curve so obtained

$$\displaystyle{(\tilde{\tau }_{1},\tilde{p}_{1}),(\tilde{\tau }_{2},\tilde{p}_{2}),\cdots \,,(\tilde{\tau }_{m},\tilde{p}_{m})}$$

is called a constant maturity forward curve. Note that except for some exceptional cases, the prices $\tilde{p}_{i}$ are the results of data analysis, and they are not observed quotes from the market. So any conclusion drawn from the analysis of these modified prices is subject to artifacts created by the way we massaged the data, and should possibly be taken with a grain of salt!

Figure 5 gives the plot of the daily forward curves for Copper between January 3, 1990 and July 7, 2013. The trading days t appear on the axis labelled “Date” while the resampled time to maturity appears on the axis labelled “Tau”. We express τ in months and in the particular case of Copper, we resampled the continuous maturity forward curve for the values τ = 1, τ = 2, ⋯ , τ = 24 months.

Fig. 5

Surface plot of the daily forward curves for copper between January 3, 1990 and July 7, 2013

Principal Component Analysis (PCA) is the most basic data analysis technique to identify the effective dimension of multidimensional objects. It was successfully used by Litterman and Scheinkman in [24] to identity the main factors in the time evolution of the term structure of interest rates. Since then, it has been used systematically each time a financial engineer faces a forward curve of any kind. We performed PCA on the daily changes in the Copper constant maturity forward curves over two different periods, period P ₁ ranging from January 3, 2000 to December 31, 2004, and period P ₂ ranging from January 3, 2010 to July 7, 2013 (Fig. 6).

Fig. 6

Proportions of the variance explained by the loadings of the PCA of the copper forward curves for the period P ₁ ranging from January 3, 2000 to December 31, 2004 (left) and the period P ₂ ranging from January 3, 2010 to July 7, 2013 (right)

The first four loadings of each of the PCAs are reproduced in Fig. 7. While the shapes of the first loadings are strikingly similar (the first and main one representing a parallel shift, the second one corresponding to a tilt of the curve, while the third one provides convexity or concavity to the curve), the proportions of the variance explained by the factors which are given in Fig. 6 deserve some explanation. Despite the fact that the scales of the vertical axes partially mask the differences between the two periods, it appears clearly that the term structure of forward prices is stiffer in the second period. By this we mean that a smaller number of factors explains the same proportion of the fluctuations in the time evolution of the forward curves. This phenomenon is widespread throughout the commodity markets and seems to have appeared in the mid-2000s. More on that later on.

Fig. 7

Loadings of the PCA of the copper forward curves for the period P ₁ ranging from January 3, 2000 to December 31, 2004 (left) and the period P ₂ ranging from January 3, 2010 to July 7, 2013 (right)

2.6 Market Participants

The original raison d’être of the commodity markets was to facilitate price discovery and allow the transfer of price risk from producers and consumers to agents willing to assume that risk. Over the last decade, the growing financialization of these markets has dramatically changed this idealized picture, and the activity of these markets became increasingly murky and time and again more difficult to compartmentalize.

In its weekly Commitment of Traders (COT) reports, the CFTC provides information on the various categories of market participants which are active in the commodity markets. Originally, these participants could be organized in two major groups: hedgers trading in futures contracts to reduce an existing risk exposure in their commercial business (which is the reason why they are also called commercials), and speculators or non-commercials. However, through financialization, an increasing number of commodity index swap dealers who hedge to offset financial positions were categorized as commercials. To remedy this problem, starting in 2007, the CFTC added a supplementary Commodity Index Traders (CIT) report, and more recently, weekly Disaggregated Commitment of Traders (DCOT) reports. The five categories of market participants identified by the DCOT reports are given in Table 1. The reader interested into more details is referred to [35].

Table 1 CFTC classification of commodity markets participants from its “Disaggregated Commitment of Traders Reports”. See [35] for details

The swap dealer category is not limited to passive investors tracking commodity indexes. It includes swap dealers who do not have commodity index-related positions. On the other hand, money managers trade on short-term horizons and adopt active investment strategies.

2.7 Exchange Traded Products and Index Investing

Exchange traded products (ETPs) include exchange traded funds (ETFs), exchange traded vehicles (ETVs), exchange traded notes (ETNs) and exchange traded certificates (ETCs). Many energy or commodity ETFs are tracking proprietary benchmark indexes measuring the aggregate performance of stocks in the energy or commodity sector. For example, the Vanguard Energy ETF (VDE) is a typical passively managed portfolio aiming at a full replication (whenever possible) of a portfolio of stocks of companies involved in the exploration and production of energy products such as oil, natural gas and coal. Most ETPs replicate the return on a single commodity, or a group of commodities. ETPs issue shares that are traded like a stock on a securities exchange. So the shares of ETPs are traded on equity markets. Some of them are easily accessible by small-scale investors, while others offer large single coupons, and are therefore more attractive to institutional investors such as pension funds. Apart from ETFs for precious metals, such funds have traditionally used futures contracts as collateral. But an important recent development is that some ETPs, such as those in copper and aluminum, are now backed by physical commodities. Futures-backed ETPs expose investors to counterparty risk, as transactions involving buying and selling of ETPs do not go through a clearing house on commodity exchanges. The rising importance of physically-backed ETPs indicates that risk aversion and growing concern with counterparty risk have made it more acceptable for financial investors to bear the storage cost of the physical commodities as they can be used as collateral. The currently very low interest rates, which reduce the cost of credit used to finance storage costs, has most likely also contributed to the increased importance of physically-backed ETPs. Returns on such products are determined by spot price movements, while the returns on futures-backed ETFs are largely influenced by the roll yield, and thus share the characteristics of traditional index investments.

2.7.1 ETVs

Exchange Traded Vehicles (ETVs) provide investors exposure to commodity futures contracts without actually trading futures or ever taking physical delivery of the underlying commodity. Most often, they track a single commodity, as opposed to an index computed on a portfolio of commodities. They are traded as equities on equity markets. They can be short-only as well as long-only

2.7.2 ETNs

Exchange Traded Notes (ETNs) are debt securities issued by banks. Up until they mature, their returns are based on the performance of an underlying index. They combine features of bonds and ETFs. ETNs’ values are affected by the credit worthiness of the issuer. As a result, their values depend not only upon the value of the underlying portfolio of commodity contracts, but also on the credit rating of the issuer.

2.7.3 Commodity Mutual Funds vs ETFs

The reason for broad commodities mutual funds’ popularity, say professional investors, is largely due to the fact that similar commodities ETFs hold futures contracts. This leaves ETFs more prone to so-called contango effects, as well as vulnerable to tax hits and front-running. As already mentioned earlier, precious metals ETFs, however, avoid these problems by directly owning their underlying commodities.

There are two types of commodity ETFs. Those which track an index computed from the performance of a portfolio of stocks of companies whose business is commodity intensive, and those which track the performance of a commodity index. We are mostly concerned with the latter. They usually hold futures contracts because the definitions of the indexes they track are based on the performances of specific contracts. But this can lead to problems, as the ETFs have fallen victim to contango when a fund loses money every time it rolls over from a near-month contract to a further-dated contract. See the example of UNG discussed below.

Some mutual funds, case in point PIMCO Commodity Real Return Fund PCRDX, have tried to avoid these pitfalls. Their strategy is to gain exposure to commodities through debt instruments such as swaps and pre-paid forward notes, rather than futures, in order to avoid the hit of a normal backwardation/congango transition.

2.7.4 Index Investing

The final way to gain exposure to commodity which we discuss in this chapter is investing directly in Commodity Indexes or in ETPs tracking these commodity indexes. For liquidity reasons, most ETPs simply invest in contracts with the shortest possible times to maturity. When the contracts they are holding approach maturity, in order to avoid delivery or settlement issues, they automatically roll their holdings by closing the positions in the contracts approaching maturity, and taking the same exact positions in the contracts available for trading with the shortest possible maturities. See the discussion of the example following the definition of roll yield in Sect. 2.4 and of the roll algorithm in Sect. 3.5 below. This form of passive investment (after all there is no need for a Commodity Trading Advisor (CTA) for that), has become very popular as a way to diversify an investment portfolio with an exposure to commodities without having to deal with the gory details of all the convoluted idiosyncrasies of the relevant markets. Nevertheless, an understanding of forward curve dynamics and the effect of frequent (typically monthly) rolls is still vital, as a recent investor in a natural gas ETF would undoubtedly agree: between June 2008 and March 2012 this ETF (called UNG) lost a shocking 96 % of its value, with roughly half attributable to the spot price drop and half to the steep contango witnessed throughout this period.

However, one the main original contribution of this chapter, if any, is to review and investigate the impact of a new generation of commodity indexes with a different roll mechanism, optimizing the roll yield. See Sect. 5 below for details.

2.8 Active Versus Passive Investing

Investing in a portfolio tracking the composition of an index like those discussed in Sect. 3 below, is often called indexing. It is a form of passive investing because managing such a portfolio does not require active involvement, except for setting up the portfolio and periodic re-balancings. The expected performance of indexing is no different from the performance of the benchmark index. This is in contrast with active investing whose objective is to outperform the market or a benchmark index. Depending upon their styles, active managers rely on fundamental analysis, technical analysis or macroeconomic analysis to identify inefficiencies and anomalies in the markets which they then try to exploit. In a recent report [33], the United Nations Conference on Trade and Development (UNCTAD) argued that between July 2009 and February 2011, the importance of index traders diminished at the expense of active investment strategies. Based on Bloomberg and CFTC data, it published Pearson correlation coefficients between prices in specific commodities (e.g. oil, cocoa, maze, sugar and wheat) and positions in these commodities by index investors on one hand, and money managers on the other hand. These numbers show a close correlation between commodity prices and the positions of financial investors that pursue an active trading strategy. See also the shorter and more aggressive policy brief [34] mostly focused on WTI Crude oil prices.

3 Commodity Indexes

Indexes can be traded through the use of index swaps which involve the exchange of a fixed payment for the value of the index at a pre-determined date. In most cases, this type of passive investment relies on ETPs, such as ETFs, backed by portfolios of futures contracts more often than individual futures contracts. The commodity-related assets under this form of management was at a historic high in March 2011, when it reached about $410 billion which is approximately the double of its pre-crisis level of 2007. While index investment accounted for 65–85 % of the total between 2005 and 2007 prior to the financial crisis, its relative importance fell to 45 % since 2008. This decline occurred despite a roughly 50 % increase in the value of index investments between 2009 and the end of 2010.

3.1 Index Terminology

We now give specific definitions for some of the terms we already used when we commented on some of the figures at the beginning of the chapter.

• A Spot Index is based on the prices of the contracts included in the index;

• An Excess Return Index incorporates the returns of the corresponding spot Index as well as the discount or premium obtained by rolling hypothetical positions in contracts approaching their delivery dates;

• A Total Return Index incorporates the returns of the corresponding excess return index as well as the interests earned on fully collateralized contract positions on the commodities included in the index.

As for the original indexes introduced in Table 2, we briefly review the main features of CCI and RICI in Sects. 3.2 and 3.3 respectively, and we postpone the discussion of the major indexes GSCI and DJ-AIG to Sect. 3.4.

Table 2 Original commodity indexes

3.2 The Continuous Commodity Index (CCI)

The Continuous Commodity Index has been around since 1986 as a means to track the overall performance of the commodity markets and to offer investors a way to trade a diversified group of commodities under one contract. CCI is a broad grouping of 17 different commodity futures. It is one of many reincarnations of the original CRB Index that was developed in 1957. It is equally weighted. Each member commodity represents 5. 88 % of the index. Over the years, some commodities have been dropped and replaced by new ones to give a better representation of the overall performance of commodities.

For the sake of completeness, we list by groups the commodities currently included in the Continuous Commodity Index CCI (Table 3):

Table 3 CCI composition

Notice that this partition of the index universe into commodity groups does not coincide with the partition given in Table 5 of the universes of the S&P-GSCI and DJ-UBSCI indexes into sectors. This is unfortunate, but typical of the historical lack of standardization of commodity indexes which change over time.

3.3 The Rogers International Commodity Index (RICI)

This total return index was designed by James B. Rogers, Jr. in the mid 1990s. It comprises futures contracts on 36 physical commodities ranging from agricultural to energy and metals products, quoted in four different currencies, listed on 12 exchanges in five countries. Its goal is to capture the price of raw materials throughout the world, and consumption patterns in developed as well as developing economies.

Over the past decade, two commodity indexes have emerged as industry behemoths: the Standard and Poor’s-Goldman Sachs Commodity Index (S&P-GSCI), and the Dow Jones-UBS Commodity Index (DJ-UBSCI). They are marketed as tradable and for this reason, they are based on liquid commodity contracts traded on highly active futures markets.

3.4 The Two Major Commodity Indexes

In this section, we present the two major commodity indexes: the S&P-Goldman Sachs commodity index (SP-GSCI) and the Dow Jones-UBS commodity index (DJ-UBSCI). While they have been historically fierce competitors, The McGraw-Hill Companies owning the S&P indices and the CME Group, major shareholder in Dow Jones Indexes merged in the summer of 2012 to form the giant index provider S&P Dow Jones Indices.²

3.4.1 The Dow Jones-UBS Commodity Index

Introduced on July 14, 1998, as the Dow Jones-AIG Commodity Index, this index is rebalanced annually, the weights being based on production and liquidity as long as, after each rebalancing, no commodity group constitutes more than 33 % of the index, and no single commodity constitutes more than 15 % of the index. It was acquired in May 2009 by the Swiss bank UBS AG.

3.4.2 The S&P-Goldman Sachs Commodity Index

Goldman Sachs published the GSCI starting 1991. It was acquired by S&P Indices in February 2007 when it became the SP-GSCI. The weights used in the computation of the index value are based on world production of the physical commodities. When a futures contract included in the index approaches maturity, a smoothed rolling procedure is implemented to replace the soon to mature contract with the next to nearest maturity contract. As most commodity indexes, it comes in three flavors: Excess Return, Total Return, and Spot. A time series plot of the spot index was given in Fig. 1.

3.4.3 Comparison

Table 4 provides a detailed comparison, as of August 2013, of the compositions of the two major commodity indexes.

Table 4 Side by side comparison of the two major commodity indexes

Table 5 provides a summary comparing the weights given by the two indexes to the various commodity sectors.

Table 5 Sector by sector comparison of the two major commodity indexes

3.5 The Roll Algorithm

While the composition of equity indexes can be relatively stable, commodity indexes have to deal with the issue of maturing contracts. Even if the relative proportions between the physical commodities entering an index can remain stable over time, futures contracts approaching maturity need to be replaced by similar contracts with longer maturities in order to avoid to have to take delivery of the physical commodities. Each index prospectus describes the algorithm used to roll the contracts approaching maturity into longer lived contracts. For the most part, the indexes considered in the first part of this chapter use a simple roll strategy, replacing contracts nearing delivery by contracts with the next maturity dates. There are some exceptions, due mostly to liquidity considerations. These exceptions are spelled out in documents publicly available, but the index boards reserve the right to alter the rolling procedures on a case by case basis when exceptional market conditions render the rolling algorithm unpractical.

As mentioned several times already, when the forward curve is in backwardation, replacing a maturing contract by the nearest maturity contract results in a net gain which is called the roll yield. However, when the curve is in contango, rolling contract is done at a cost. This simple fact needs to be kept in mind when one think about investment in commodity futures.

4 Review of the First Wave of Works on the Financialization Hypothesis

In [31], Singleton uses data from the 2008 boom—bust in oil prices to argue that flows from institutional investors have contributed significantly to the volatility of commodity prices.

In a decisive study [32], Tang and Xiong refute the idea that growing demand from emerging economies was the only driver of the commodity price burst in 2006–2008, and that commodity prices were influenced by financial factors and financial investor behavior. They use correlation coefficients computed in a trailing sliding window to argue that the co-movements between oil and other commodities rose dramatically following the inflow of institutional investors starting from 2004. Comparing with non-index commodities, they also demonstrate that this correlation increase effect is especially pronounced among commodities included in the same indexes. They show that the co-movements of the prices of different commodities increased after 2003–2004, and argue that this coincides with the beginning of significant position-taking by commodity index investors. A further evidence of that claim is the fact that for the commodities included in the major indexes this increase was significantly greater than for those not included.

We first illustrated the dramatic increase in return correlations between equities (as represented by the S&P 500 index) and commodities (as represented by the GSCI Spot index) in Fig. 2. There we can clearly see the increase in the instantaneous “beta” over the period 2006–2009. We further stress this claim by reproducing in Fig. 8 the time evolution of the instantaneous “betas” of Copper against the S&P 500. As expected this plot is noisier since we lost the averaging effect of the commodity index, but it is still providing a strong evidence for the tightening of the correlations between commodities and equities over that period.

Fig. 8

Instantaneous dependence (β) of copper daily returns upon S&P 500 returns

Based on a thorough analysis of a proprietary dataset from the CFTC [5], Buyuksahin and Robe argue that the recent increase in the correlation between equity indices and commodities is due to the presence of hedge funds active in both equity and commodity markets.

In a recent study [15], Henderson, Pearson and Wang show that large investments in Commodity Linked Notes (CLNs) are the sources of hedges which cause significant price changes in the underlying futures markets.

However, not all the evidence point in the same direction. Surveys by Irwin and Sanders [16], and Fattouh, Kilian, and Mahadeva [21] argue against the claim that increased speculation in oil futures markets was an important factor in oil prices evolution. Furthermore, Kilian and Murphy [20] argue that the 2003–2008 oil price surge was due to global demand shocks rather than speculation. See also [26] and the technical report from the European Central Bank [25] for more balanced conclusions.

Following Kyle and Xiong [23], one can argue that portfolio rebalancing of commodity index funds can lead to correlated trades in related markets and thus create spillover effects across different commodities. In a recent econometric study of agricultural commodities, Hamilton and Wu [14] found no evidence that the positions of traders identified by the CFTC as index traders can help predict returns on the front month futures contracts.

While there is still lack of agreement on whether institutional investors affect commodity futures prices, it is well-established that institutional investors trades do affect stock prices. In the case of equity markets, several studies have analyzed the so-called asset class effect according to which correlations between assets belonging to the same index are higher than those between index and non-index assets. The co-movements associated to these unusually high correlations are attributed to the presence of institutional investors. This type of analysis was extended in [1] with an attempt to incorporate some of the idiosyncrasies of the commodity markets.

5 A New Generations of Indexes

Returns from investing in commodity futures contracts come typically from three different sources: spot, collateral and roll returns. New generations of indexes have chosen to optimize the roll return which was traditionally left to the backwardation/contango transitions. Among the most successful of this new breed of indexes, the Deutsche Bank Optimum Yield Commodity Index rolls according to a formula rather than simply rolling month to month. The formula seeks to achieve the best roll return possible given the shape of the forward curve at the time of the roll. Instead of rolling a contract nearing maturity into the nearest available maturity contract, the roll algorithm chooses the maturity with the best implied annual roll yield, as long as some liquidity constraints are satisfied.

We shall speculate on the possible impacts of the tracking of such indexes in our further look at consequences of this form of financialization.

5.1 Deutsche Bank PowerShare Optimum Yield Commodity Index

This Index comprises futures contracts on 14 heavily-traded physical commodities. with a distribution target of 55 % energy, 10 % precious metals, 12.5 % base metals, and 22.5 % agriculture. The weights are computed according to a combination of production and market liquidity. It is rebalanced annually in November. We give below a snapshot of its composition. The main originality of this index is the process used to implement the roll. As a general rule, commodity futures-based indexes replace contracts before they expire, and automatically buy into the next available maturity month. As explained earlier, this process is called “rolling” futures contracts forward. Instead of following this common practice, PowerShares DB Commodity Optimum Yield Index (and the ETFs tracking these indexes) use a procedure which is called Optimum Yield Roll process. As described in public prospectuses, it consists in choosing the maturity month among the next 13 maturity months available for trading at the time of the roll, which offers the best possible roll yield. As a result, the maturities of the futures contracts used in the computation of the PowerShares DB Commodity Index are not limited to the nearest month (Table 6). Accordingly, the portfolios of the corresponding Tracking Funds includes contracts with maturities further down the curve. While the details of the roll algorithm remain somehow mysterious due to the liquidity factor coming into the choice of the maturities to roll into, this roll strategy has been credited for out-performing the major indexes, both the SP GSCI and the Dow Jones-UBS commodity indexes, in the period 2006–2009 (Fig. 9).

Table 6 Composition of the DB iShare index as of 09-Aug-2013 12:00 AM

Fig. 9

Time series plot of the daily GSCI total return index (black) and Deutsche bank iShare DBiS (red)

The commodities included in the index are traded on the following futures exchanges:

• NYMEX: Light Sweet Crude Oil (WTI), Heating Oil, RBOB Gasoline and Natural Gas;

• ICE Futures Europe: Brent Crude;

• Commodity Exchange NY: Gold and Silver;

• London Metal Exchange: Aluminum, Zinc and Copper Grade A;

• Chicago Board of Trade: Corn, Wheat and Soybeans;

• ICE Futures U.S.: Sugar.

5.2 Dow Jones-UBS Roll Select Commodity Index

Deutsche Bank is by far not the only financial institution to have tried to capitalize on the attractiveness of the roll yield optimization. Indeed, a version of the Dow Jones-UBS Commodity Index was designed with the same goal in mind. Its goal is to mitigate the effects of contango on index performance. For each commodity included in the index, the roll algorithm chooses the futures contract (within the next nine maturity month available), which exhibits the most backwardation or least contango.

Table 7 Sector distribution of the UBS Bloomberg commodity indexes

5.3 The UBS Bloomberg Constant Maturities Commodity Index

Partly motivated by the losses incurred by the traditional indexes in the recent contango period, the UBS Bloomberg Constant Maturity Commodity Index (CMCI) uses constant maturity contracts to provide diversification across maturity dates.

While the distribution of the relative weights across the sectors is not much different from the major commodity indexes, the goal of the index is to overload the diversification across the 28 commodities included in the index by a diversification across five constant maturities τ = 3mo, τ = 6mo, τ = 1Y, τ = 2Y and τ = 3Y, with weights varying with the commodities. The details are given in Tables 7 and 8.

Table 8 Composition of the UBS-Bloomberg constant maturities commodity index (Target weights H1-2013)

5.4 Still More Commodity Index Rolling Down the Curve

The Credit Suisse Commodity Benchmark (CSCB) index is also a long-only index of commodities weighted by world production and liquidity. It is rebalanced monthly, and contracts approaching maturity (starting 15 days prior to actual maturity) are rolled into equally weighted averages of the three contracts with maturities up to 3 months further out the term structure of forward prices.

Barclays has also a suite of exchange traded products tracking commodity indexes based on portfolios of futures contracts updated with optimized rolling algorithms. Among its many ETPs, the Barclays Capital Commodity Index Pure Beta TR and Barclays Capital Commodity Index TR are ETNs (iPath Pure Beta ETNs) tracking commodity indexes created by Barclay implementing a rolling algorithm involving varying expiration dates, typically choosing at roll time, the contract with the highest positive implied roll yield when the curve is backwarded or the lowest negative return when the forward curve is in contango.

We claim that the presence of these funds pushed the open interest down the curve, phenomenon which we now demonstrate in Sect. 6 devoted to a discussion of the impact of the financialization of commodities on open interest data.

6 Commodity Open Interest

So far, our discussion has been mostly concentrated on prices. We switch gear and turn our attention to two important variables whose values and changes can shed informative light on the future evolutions of prices. The first of these variables is volume. On any given day, and for each contract maturity, volume quantifies the trading activity in this particular contract. It provides a measure of the amount of contracts that have changed hands, the amounts of new positions open or closed for this specific maturity date. While a good indicator of the volatility of the market, it may not be as representative of economic fundamentals as it is of trader sentiments and behaviors. We choose to study open interest instead. On any given day, and for each contract maturity, open interest is the total number of outstanding contracts with that specific maturity that are held by market participants on that day. These numbers are often aggregated over the set of all maturities available for trading and a total open interest figure is given as the total number of outstanding contracts held by market participants on that day. We used this aggregate open interest for Crude Oil earlier in the chapter (recall Fig. 4) to illustrate the influx of investments over the period 2004–2009. We give the corresponding plot for Copper in Fig. 10 below.

Fig. 10

Time series plot of the daily open interest in copper futures contracts between January 2, 1990 and September 9, 2013. Source: Data Stream

Fig. 11

Surface plot of the daily term structure of open interest for WTI crude oil (left) and copper (right) futures contracts between January 3, 1990 and July 7, 2013

6.1 The Term Structure of Open Interest

The purpose of this section is to demonstrate the changes in the term structure of open interest which occurred in the mid 2000s. Our contention is that open interest slid down the curve as investment in longer maturity contracts increased. We illustrate these claims with a close look at the two commodities we followed throughout the chapter: WTI crude oil and copper. While crude oil may have a seasonal component, it is not strong enough to overwhelm the features we are looking for. The same analysis would have been more difficult with natural gas.

Figure 11 gives the surface plots of the term structures of open interest for WTI crude oil and copper. The plot in the left pane clearly shows that the highest open interest is concentrated on the shortest available maturity (the variable Date being close to 0), and that for longer times to maturity, a secondary bump appears. However, the time evolution of the location of this bump shows a clear shift further down the curve in the mid 2000s. A similar phenomenon, though not as clean because of noise, can be observed in the right pane in the case of copper.

In order to provide one more graphical evidence of the open interest slide down the curve, we performed the PCA of the daily term structure of open interest over the two time periods considered so far. The results are reproduced in Fig. 12. Contrary to the daily changes in price, it appears that more factors are needed to explain the fluctuations over the second period. But looking at the loadings plotted in Fig. 13, we clearly see a shift to the right of the bumps representing where most of the open interest is expected.

Fig. 12

Proportions of the variance explained by the loadings of the PCA of the open interest copper forward curves for the period January 3, 2000 to December 31, 2004 (left) and the period P ₂ ranging from January 3, 2010 to July 7, 2013 (right)

Fig. 13

Loadings of the PCA of the open interest forward curves of Copper for the period January 3, 2000 to December 31, 2004 (left) and the period ranging from January 3, 2010 to July 7, 2013 (right)