Permanent and transitory price shocks in commodity futures markets and their relation to speculation

Abstract

This paper takes an innovative look at the relationship between commodity futures prices and speculation. Contrary to other studies, we analyze the effect of speculation on temporary and permanent futures price shocks estimated from a cointegrated system of pairwise short- and long-dated contracts. Where cointegration is found, the long-term equilibrium is determined by the long-dated contract, while the adjustment toward equilibrium is restored by the short-dated contract (except for cotton). Granger causality tests cannot reject the null hypothesis that speculation as measured by Working’s T index has no effect on squared permanent price shocks for 7 out of 9 commodities. Where the null hypothesis is rejected, the relationship exhibits a negative sign, i.e., speculation has a stabilizing effect.

1 Introduction

In recent years, commodities futures trading has repeatedly be blamed for destabilizing spot markets of commodities, in particular for the increase in many spot prices or their volatilities. While the academic and public debate covers an extremely wide range of topics such as the role of speculation in general, the emergence of long-only investment products (e.g., index investing), or structural demand-side effects (e.g., biofuels), this paper addresses a rather specific issue related to the price discovery process of futures markets and its relation to speculation.

The distinguishing feature of this paper is the focus of the effects of speculation on permanent and transitory (P–T) price shocks. Thus, we analyze price levels, not returns and their variability. This is accomplished by extracting the innovations from a cointegrated system of long- and short-dated futures contracts which are transformed to P–T shocks for analyzing the effects of speculative activity.

The discrimination between transitory and permanent price shocks is important: It makes a difference whether speculation distorts the price discovery process, i.e., the process for establishing the long-run price equilibrium, but without affecting the equilibrium, or whether it permanently shift the long-run price level of commodities. Our P–T decomposition thereby exploits the diverging price behavior of short- and long (medium-term)-dated futures contracts for a sample of 14 liquid commodities traded in the USA.

Interestingly, the current public discussion does not put much emphasis on the distinction between temporary and permanent price impact of speculation in commodity markets—should it be observed at all. This is in contrast to the debate about speculative price effects back in history, e.g., in the early 20th century, after the German Exchange Bill restricted or even prohibited futures trading on certain grains at the Berlin Produce Exchange. This regulation was the political result of an ongoing controversy between academics and agriculturists about the impact of speculation on declining commodity prices and higher volatilities. Most academics argued that speculation does not affect the long-run level of commodity price because adjustments in physical supply (e.g., increase in acreage) and demand (e.g., improvement in feeding technology) would offset these effects, but might accelerate daily or at best weekly price changes.¹ However, the statistical techniques were not available in these days to estimate the persistence of price changes.² Today, these techniques are well developed, but they are not used to contribute to the ongoing debate about the price effects of speculation.

Traditional tests in the empirical literature study the price effects of speculation by using those futures contracts where speculation is assumed to be most active, i.e., short-dated maturities. Unfortunately, the statistics about commodity speculation and hedging³ are not publicly available across the maturity spectrum of futures contracts,⁴ which makes it impossible to provide direct evidence about the maintained claim. The approach in this paper is different: We construct P–T decomposed price shocks and then analyze their relation to speculation. This makes it possible to find out whether speculative activities are related to permanent or transitory price shocks, or none.

Our hypothesis about the impact of speculation on commodity prices is tested in two consecutive steps:

In a first test, we analyze the price discovery process of short- and long-dated futures contracts by testing for cointegration and estimating a vector error correction model (VECM). Intertemporal arbitrage implies that short- and long-dated futures contracts on the same commodity reflect the same underlying fundamental information and should therefore be cointegrated.⁵ However, frictions (e.g., transportation and storage costs, scarcity or limited warehouse capacity) and market imperfections due to diverging demand and supply for hedging and speculation across maturities may have diverging temporary or permanent pricing effects on long- and short-dated contracts.⁶ The residuals of the VECM are used to estimate these (possibly) diverging effects. Specifically, the technique of Gonzalo and Ng (2001) is applied to disentangle permanent and transitory shocks driving the system of futures prices.

A second test addresses the temporal relationship between the permanent and transitory shocks of the cointegrated system and the net amount of speculation as measured by Working’s T index by using Granger causality tests.

The empirical results reveal two major findings: First, where cointegration is found, the long-term equilibrium is determined by the long-dated contract, while the adjustment toward equilibrium is restored by the short-dated contract (except for cotton). Under the assumption that speculation occurs mainly in short-dated contracts, our results are inconsistent with the hypothesis that speculation distorts the process for establishing the long-run price equilibrium. Second, Granger causality tests cannot reject the null hypothesis that speculation has no effect on squared⁷ price shocks for most of the analyzed commodities. These findings hold for permanent as well as transitory price shocks in 7, respectively 8, out of 9 cases. In the few cases where the null hypothesis is rejected, the relationship exhibits a stabilizing effect.

The rest of the paper is organized as follows: Sect. 2 reviews some of the research topics related to speculation and index trading in commodity markets which are relevant for our study. Section 3 outlines our empirical methodology and test procedure, and Sect. 4 contains a detailed description of the data sources and the construction of our time series. The results of our tests are presented in Sect. 5, which are summarized in Sect. 6.

2 Speculation and index trading: a review of selected research topics

The extent and role of speculation in commodity (and foreign exchange) markets have always been debated in the commodity futures literature, even back in the 19th century when the first organized markets for forward and futures contracts were launched. The vast literature is reviewed in various surveys.⁸ The current debate is triggered, first, by the drastic increase in several commodity prices (for example, major food prices more than doubled within a short time period, from 2005 to mid-2008), second, by the growth of index-related commodity products, and finally by the pressure on institutional investors to respect or to define standards about responsible investing such as the UN-supported “Principles of Responsible Investments” initiative. These developments have accelerated new empirical research on commodities, with new methodologies, new data, or a new focus.

A major focus on recent empirical studies is the impact of commodity index investing on spot and futures prices. Index trading, i.e., investing in a synthetic basket of commodities which tracks a specific commodity index⁹ via OTC swaps or structured products, became popular and was rapidly growing in the 2000s. The indices represent a rollover strategy in long positions of short-dated futures contracts on the underlying commodities; most indices are heavily weighted in energy-related commodities (typically between 60 and 70%), mainly crude oil. Since index investors hold net long positions in the underlying futures,¹⁰ it is argued that the net long positions of commodity index investors push prices of certain commodities to levels beyond those justified by fundamentals. The argument is reinforced by the observation that unlike traditional speculators, index investors—by rolling the positions forward—take effectively long-term positions in the underlying risks. Index investing is also seen as one of the major causes of the increased volatility of non-energy commodity prices as well as their increasing correlation with the price of oil, which is often called “financialization” of commodity markets. The criticism is not only advanced by prominent investors (for example Soros 2008 or Masters 2008), but can also be found in official reports of the US Government¹¹, international organizations (e.g., UNCTAD 2011) as well as in pamphlets of special interest groups such as the World Development Movement or research institutions, e.g., the International Food Policy Research Institute (Cooke and Robles 2009). Notable contradictions are advanced in reports commissioned by the IOSCO (2009), the OECD (Irwin and Sanders 2010), or the World Bank (Baffes and Haniotis 2010).

The financialization argument is well documented and discussed by Baffes and Haniotis (2010); the authors observe a strong impact of energy prices on the prices of non-energy commodities with an increase during the recent boom. Even though they conclude that “index fund activity (one type of ‘speculative’ activity among the many that the literature refers to) played a key role during the 2008 price spike,” they do not provide any direct test addressing the role of index trading, open positions of speculators, or the like to underpin their argument. More direct to that point is a study by Tang and Xiong (2012). They analyze daily returns of nearby non-energy commodity futures from 1998 to 2011 and find that the increase in correlations with oil price changes after 2004 is “significantly more pronounced for indexed commodities than for off-index commodities after controlling for a set of alternative arguments” (p.54), what the authors interpret as explanation of the increased volatility of non-energy commodity prices. An econometrically more rigorous analysis of index trading can be found in Gilbert (2010). The author finds strong evidence that the correlation between the changes of oil and food prices is the result of “common causation and not of a direct causal link.” Granger causality tests reveal that index investing qualifies as a channel through which macroeconomic and monetary factors may affect the increase in food prices. A limiting factor of this analysis is that it is based on food and non-food price indices, rather than on individual commodities’ prices.

Notable counterarguments of the index-based explanations of commodity price increase and volatility are the following:¹² The first and most immediate argument is that price increases over the past decade can also be observed for commodities on which no futures contracts are traded and which are not constituents of commodity indices.¹³ Examples include cadmium, rhodium, cobalt, or coal. Second, speculation can only affect commodity prices insofar as it affects inventories. If speculation drives commodity prices, then it pays for speculators to increase inventories—but this is the contrary to what is observed in the recent boom: inventories are low. Gilbert (2010) argues that it takes time for this effect to materialize because the incentive to hold inventory is driven by long-dated futures prices, and there is a time lag in production and consumption decisions (and hence, inventories) to respond to higher prices. Of course, whether this is true or not is an empirical question. Third, valid speculation proxies for commodity futures markets should be measured relative to the amount of hedging. Specifically, Working (1960) proposed an index (called T index) which highlights the relative amount of net excess (long or short) speculation with respect to the hedging positions of commercial traders. Sanders et al. (2010) use data from the CFTC Commitments of Traders (COT) report and the Commodity Index Traders (CIT) report¹⁴ from 1995 to 2005 and 2006 to 2008 to construct this index and to re-examine the amount of speculation, in particular related to index investing, for 9 non-energy commodities. For most commodities, the authors find that “the increase in long speculative positions was equaled or surpassed by an increase in short hedging.” In simple terms, more speculation is compensated by more hedging. This is not a surprising finding for markets where short hedging typically dominates. This is in line with earlier studies which apply the Working’s T index and conclude that speculation is insufficient on most agricultural futures markets relative to the hedging demand.¹⁵

Unfortunately, the standard COT data are highly aggregated across different categories of commercial and non-commercial users. In order to analyze the relationship between speculative positions and the pricing of commodities in more detail, the CFTC recently made disaggregated data available from its large trader reporting system (LTRS) to a few researchers. These data allow to analyze futures positions for many trader sub-categories (swap dealers, insurance companies, hedge funds, producers, trading advisors, etc.,) for a broad range of futures contracts, on a daily basis, and for individual maturities. A description of the data can be found in e.g., Brunetti and Büyükşahin (2009). The following results are reported in the papers available to date: Büyükşahin and Harris (2011) analyze the crude oil futures market and find that Working’s T index increased in parallel with crude oil prices from 2004–2009. Till (2009) reports similar results for crude oil, heating oil, and gasoline futures for the period 2006–2009. Brunetti and Büyükşahin (2009) perform Granger causality tests between disaggregated futures positions of three groups of speculators (swap dealers, hedge funds, and floor brokers) and futures prices in the 2005–2009 period. In contrast to other studies, their VAR test allows for interactions between the traders’ positions on returns and volatilities. They find that speculation does not cause price movements, reduces risk and thus hedging costs. Aulerich et al. (2010) use the LTRS database to analyze the price impact of long-only index funds in 12 commodity futures markets from 2004 to mid-2008. As other studies, they assume that index trading is reflected in the positions reported by the sub-category “swap dealers.” They use Granger causality tests to examine the relationship between index trader position changes and commodity futures returns, and volatility, respectively, as reflected in the nearby and next-to-nearby maturities. Only 16% of the coefficients are statistically significant, and signs are mixed. In particular, they do not find that index positions had a greater impact on returns during their first sub-period 2004–2005 when positions were growing most rapidly. If anything, then volatility seems to have been affected by the presence of index traders in several markets.

At this stage, it is worth emphasizing that it is far from being a consensus that index trading (which comprises the activities of long-only funds with direct investment in futures markets as well as the hedging activities of OTC swap dealers) should be regarded as speculation from an economic point of view. In this paper, we agree with the contrary view advocated by Stoll and Whaley (2010) that index investing is not classical speculation: The investments are not motivated by directional bets, but by the diversification benefits they provide; moreover, index investors are typically not leveraged, but fully collateralized. On the other hand, OTC swap dealers hedge their commodity index exposure originating from customized products, which is a standard hedging activity like in the commercial business (see Szado 2011, p. 81). Consistent with that view, the position data collected by the CFTC and used in this study¹⁶ classify swap dealers’ positions as “commercial”; the long-only commodity funds are, however, classified as non-commercials, i.e., speculators.¹⁷ Therefore, and rightly, the swap dealers’ positions do not show up in our speculation index used below. However, the investment positions of funds are part of our index. If speculation (or tactical asset allocation) is indeed a substantial part of the trading activities of these funds, then this should be reflected in short-dated futures contracts and be visible in our tests.

In this paper, we use cointegration analysis for a decomposition of commodity price shocks into transitory and permanent components, which are then analyzed in their relation to speculation in the underlying commodities. Methodologically, the closest to our paper is Figuerola–Ferretti and Gonzalo (2010). The emphasis of their paper is also a separation of permanent and transitory components in commodity spot and futures prices; however, their focus is on the equilibrium relationship between spot and futures price levels and its violation due to imperfect arbitrage in the presence of convenience yields. They investigate the dynamic futures–spot price relationship for five nonferrous metals in the Gonzalo–Granger P–T framework. In contrast to this study, the focus of our paper is on price shocks derived from the VECM of long- and short-dated futures prices.

3 Methodology and hypotheses

Our analysis is based on an orthogonal decomposition of the innovations of a vector error correction model (VECM) of commodity prices into permanent and temporary (P–T) shocks, as suggested by Gonzalo and Ng (2001). We first test whether the prices of short- and long-dated futures contracts, represented by the vector $\varvec{x}_t \in \mathfrak {R}^{n\times 1}$, are cointegrated by applying the Johansen test methodology. We include one short and one long maturity in our price vector ($n=2)$, as described in Sect. 4.1.

The Granger representation theorem states that cointegration implies the existence of a vector error correction model (VECM)

$$\begin{aligned} \Delta \varvec{x}_t = \varvec{\alpha } \varvec{\beta }'\varvec{x}_{t-1}+\varvec{\Gamma } \left( L\right) \Delta \varvec{x}_{t-1}+\varvec{\epsilon }_t, \end{aligned}$$

(1)

where $\varvec{\beta } \in \mathfrak {R}^{n\times r}$ represents the r cointegrating vectors, $\varvec{\alpha } \in \mathfrak {R}^{n\times r}$ the adjustment parameters, and $\varvec{\Gamma } \left( L\right) \in \mathfrak {R}^{n\times n}$ the coefficients of lagged price changes. The residuals of the VECM are used to assess the relevance of each variable in affecting the long-term equilibrium, respectively in restoring the equilibrium. The appropriate number of lags can be determined by standard model diagnostics (we use the Akaike information criterion). If the price series are cointegrated, we proceed to the decomposition of the residuals of the VECM. As shown by Gonzalo and Ng (2001), VECM innovations $\varvec{\epsilon }_t$ can be transformed to orthogonalized permanent and transitory shocks $\varvec{\eta }_t$ in two steps: first, the Gonzalo and Granger (1995) methodology is used for decomposing the VECM innovations into permanent and transitory (P–T) shocks; in the second step, the shocks are orthogonalized using a Cholesky decomposition of an auxiliary, residual variance-covariance matrix. This leads to the moving average representation of the VECM innovations

$$\begin{aligned} \Delta \varvec{x}_t = \varvec{C}(L) \varvec{\epsilon }_t = \tilde{\varvec{D}}\left( L\right) \varvec{\eta }_t, \end{aligned}$$

(2)

where $\varvec{C}(L)$ is the standard MA coefficients of the cointegrated system, while $\tilde{\varvec{D}}\left( L\right)$ represents the “orthogonal” impulse response function of the system with respect to the (P–S) shocks, determined by

$$\begin{aligned} \tilde{\varvec{D}}\left( L\right) =\varvec{C}\left( L\right) \varvec{G}^{-1}\varvec{H} \end{aligned}$$

(3)

for each lag, where $\varvec{G}$ is the Gonzalo–Granger matrix which generates the P–T shocks, and $\varvec{H}$ is the required Choleski matrix for orthogonalizing the innovations. The exact expressions can be found in Gonzalo and Ng (2001, pp. 1531–1533). The variance decomposition of the innovations of $\Delta \varvec{x}_t$ into permanent and transitory components then follows immediately.

We proceed as follows in testing our hypotheses: First, we test for unit roots in the level of futures prices; standard ADF and PP tests are performed. If the null hypothesis of a unit root is rejected, we would not further analyze that commodity. But in our sample, the hypothesis of a unit root cannot be rejected for any futures price series.

Second, a cointegration test for each pair of futures prices (a short- and long-dated maturity contract) is performed for each commodity, based on Johansen’s trace test. If the null hypothesis of no cointegration can be rejected, the adjustment coefficients of the VECM representation of the cointegrated system indicate which of the two futures contracts, the short- or long-dated, determines the long-term equilibrium of the system and, respectively, which contract establishes the adjustment toward equilibrium.

Third, the Gonzalo–Ng variance decomposition reveals the relative contribution of the permanent and transitory shocks in the adjustment process toward equilibrium.

Fourth, the squared Gonzalo–Ng P–T shocks¹⁸ are used to perform Granger causality tests with respect to excess speculation, as measured by Working’s T index. For this purpose, a bivariate linear autoregressive model $(X_1,X_2)$ including the speculation index, $X_1=T$, and the squared P–T shocks, $X_2=\eta ^2_P$ respectively $X_2=\eta ^2_T$, is estimated:

$$\begin{aligned} X_1(t)&= \sum _{j=1}^p{A_{11,j}X_1(t-j)} + \sum _{j=1}^p{A_{12,j}X_2(t-j)+z_1(t)} \end{aligned}$$

(4)

$$\begin{aligned} X_2(t)&= \sum _{j=1}^p{A_{21,j}X_1(t-j)} + \sum _{j=1}^p{A_{22,j}X_2(t-j)+z_2(t)}, \end{aligned}$$

(5)

where p represents the maximum number of lags and matrix $\mathbf A$ includes the coefficients of the model. If the null hypothesis $A_{12,1}=\cdots = A_{12,p}=0$ cannot be rejected, then $X_2$ does not “Granger cause” $X_1$; if the null hypothesis $A_{21,1}=\ldots = A_{21,p}=0$ cannot be rejected, then $X_1$ does not “Granger cause” $X_2$.

It is well known that Granger causality does not imply anything about causation in the epistemological sense. However, in the context of an error correction mechanism between long and short futures prices, which assumes a dynamic relationship by construction, the analysis of permanent and transitory shocks derived from the VECM and their temporal relation to innovations from other variables seems to be natural extension of the cointegration approach.

Finally, a variance decomposition based on a VAR model is performed to determine the proportion of permanent and transitory price innovations caused by speculation.

4 Data

4.1 Futures prices

Our empirical work is based on futures prices of 14 commodities. The selected commodities are displayed in column 2 of Table 1 and cover the following sectors: grains, softs, meat, energies, and industrial metals. The futures contracts are traded on different exchanges, which are displayed in the third column.

Table 1 Sample of selected commodity futures contracts

Monthly futures prices are used covering the time period from January 1990 to December 2010, with a few exceptions (see column 4). Shorter time periods for some commodities are due to a lack of a sufficient number of adequate maturities. The number of months covered by the respective sample periods is displayed in column 5. A breakdown of the expiry months can be found in the last column.

For each commodity, we select two contemporaneous futures prices with a short and a long delivery date. We have selected the next-to-nearby contract as short maturity and the one-year ahead of the nearby contract as long-dated contract. Of course, the long-dated contract is not really long, but it is the longest common maturity across our commodities which is available over a sufficiently long historical period. An annual maturity (or a multiple of it) is also advantageous with respect to adjusting for seasonal effects.

Futures price series are constructed applying a rollover procedure familiar in the empirical literature; details are described in “Appendix A.1.” Notice that with regard to the Granger causality tests, the futures price data must be exactly matched with the relevant reporting dates of the COT data.

4.2 Measurement of speculation: Working’s T index

Our empirical work is based on the Working’s T index which measures the degree of speculative activity. Working (1960) argued that an adequate measurement of speculation in futures markets should be net of hedging demand. He therefore suggested an index which indicates speculation in excess of those positions necessary to absorb the hedging needs. Denote speculators’ short (long) positions by SS (SL) and those of the hedgers by HS (HL); the T index is defined by

$$\begin{aligned} T= \left\{ \begin{array}{c} 1+\displaystyle \frac{SS}{HL+HS} \quad HS \ge HL \\ 1+\displaystyle \frac{SL}{HL+HS} \quad HS < HL \end{array} \right. \end{aligned}$$

(6)

depending whether there is an excess in short hedging ($HS \ge HL$) or long hedging ($HS < HL$). Intuitively, if there is short hedging pressure in a commodity (as it is mostly the case for agricultural futures), there is an economic need for long speculation for balancing out the positions. Short speculation is therefore regarded as unnecessary or excessive and put in relation to total hedging. In the case of long-hedging pressure, the T index puts (unnecessary or excess) long speculation in relation to total hedging.

Although the index measures the degree of excess speculation, it does not indicate the direction of excess speculation: The same absolute excess short and long position may lead to the same T index. Depending on the underlying economic hypothesis (e.g., the Keynes-Hicks-Cootner net hedging pressure models), we might well expect that the direction of net speculation affects the sign of the hypothesized price impact. As a consequence, the index has not so much to do with the (signed) P–T shocks themselves, but rather with their variance. Therefore, we use squared instead of plain Gonzalo–Ng P–T shocks in testing Granger causality. The use of squared price shocks makes it impossible to draw conclusions about the direction of speculative price impact, but rather about its strength.

One of the merits of the index is that it can be easily calculated from the data published in the Commitments of Traders (COT) report by the U.S. Commodity Futures Trading Commission (CFTC); details about the data used for calculating our index are reported in “Appendix A.2.” Notice that the COT data used to calculate the index, aside from traditional speculators, also include the activities of various types of long-only index funds. Since most commodity indexes represent rolling strategies in short-dated futures, our speculation index can be expected to be heavily weighted in short-term futures positions.¹⁹

Table 2 Panel A provides for each commodity the number of observations, the average as well as the minimum and maximum value of Working’s T index. Notice that only those commodities are displayed in the table which can be used in the Granger causality tests in Sect. 5.2, i.e., for which cointegration of futures prices is found in Sect. 5.1. The largest value of Working’s T index is reported for wheat (1.33), the lowest for WTI crude oil (1.07). The standard deviation of the index ranges from 0.01 for WTI crude oil to 0.07 for live cattle. Figure 1 displays four illustrative index time series: corn and wheat (first graph), WTI crude oil and live cattle (second graph). None of the series experiences an apparent structural shift within the observation period. We, however, observe substantial differences in the volatility of the series: The fluctuations of the WTI crude oil speculative index are much less pronounced than those of the other displayed series (soft commodities).

For implementing unadjusted Granger causality tests, the speculation measure must be stationary. The ADF test statistics for logarithmic index levels displayed in the last three columns of Table 2 reveal that the null hypothesis of a unit root can be rejected for all commodities in our sample. Therefore, log index levels are used for testing Granger causality.

Table 2 Summary statistics for Working’s T excess speculation index

Fig. 1

Illustrative time series of Working’s T index on net speculation. Working’s T index measures speculation in excess of these positions necessary to absorb hedging needs. The index is illustrated for corn and wheat (upper chart) and WTI crude oil and live cattle (lower chart) for monthly observations from January 1990 to December 2010. The index is calculated using position data from the Commitments of Traders (COT) report published by the CFTC

5 Empirical results

5.1 Cointegration between short- and long-dated contracts, and P–T variance decomposition

In this section, the results of the cointegration tests and variance decomposition in permanent and transitory price shocks are presented.

Stationarity: For all futures price series, the null hypothesis of a unit root cannot be rejected; depending on the commodity, a constant and/or a drift parameter is estimated.²⁰ Integrated series of the same order is a prerequisite for cointegration tests.

Cointegration: The cointegration results are displayed in Table 3. We find a single cointegration vector in 9 cases. For coffee, soybean oil, and copper, there is no cointegration²¹, while lean hogs and orange juice exhibit two cointegration vectors implying stationarity. The table shows the estimates of the restricted model²² to which we subsequently refer are shown in the second panel. The p values of Johansen’s trace test and the coefficients of the cointegration vector are displayed on the top of the table. The adjustment coefficients of the VECM are displayed below; they indicate that the long-run equilibrium for 8 commodities is driven by the long-dated contract, while the short-dated contract is responsible for the adjustment toward the trend. For one commodity, cotton, the adjustment runs in the reverse direction. Finally, the cointegration results are robust with respect to the number of lags specified in the cointegration equation.

Table 3 Cointegration of 1-month and 1-year commodity futures prices: Johansen tests

Variance decomposition: For those 9 commodities where cointegration between short- and long-dated futures prices is found, a Gonzalo–Ng decomposition of the VECM residuals is undertaken; this results in a time series with transitory shocks (TS) and a series with permanent shocks (PS) for each commodity. For illustrative purposes, the impulse response function (IRF) of the short- and long-dated corn and WTI crude oil contracts with respect to the TS and PS is displayed in Fig. 2. The IRFs confirm that the long-dated futures prices are largely driven by permanent shocks, and the effect is stable across lags. In contrast, the short-dated futures prices are mainly driven by transitory shocks. In the case of corn (WTI crude oil), the transitory shocks account for 85% (75%) of total variance of the short-dated contract, which in turn implies that the permanent effect plays a minor role at the short time horizon. As the VECM suggests, the short-dated contract is mainly responsible for restoring the equilibrium. The half life of the transitory shocks implied by the adjustment coefficient of the VECM is 20 months for corn and 16 months for WTI crude oil.

Fig. 2

Illustrative impulse response function for corn and WTI crude oil based on the Gonzalo–Ng variance decomposition. The impulse response function on the left (right) depicts the price reaction of the short- and long-dated futures contracts on a permanent (transitory) shock (P–T) by one standard deviation. The P–T decomposition is based on the residuals of the VECM (vector error correction model) representation of the cointegrated futures price system using the methodology by Gonzalo and Ng (2001). The models are estimated from monthly observations from January 1990 to December 2010

The overall results of the variance decomposition are displayed on the final row, respectively the final two rows, of each panel in Table 3. The results indicate that the transitory shocks account for 69–100% of the variance, with values close to 100% for soybean, cotton, and sugar. The half life of the transitory shocks is in the range of 10 months to 23 months. A rather quick absorption is observed for heating oil and sugar, while a long absorption period emerges for corn, wheat, live cattle, and natural gas. There seems to be no commodity-specific pattern in this reaction. Notice that the time series for sugar and natural gas differ with respect to the starting date (05/93 and 02/94 instead of 01/90).

5.2 Granger causality between squared permanent and transitory price shocks and speculation

The results of the Granger causality tests are displayed in Table 4. Recall that speculation is measured by the levels of Working’s T index which is stationary. The second and third columns of the table display the test statistics for causality between squared permanent shocks (PS) and speculation. The null hypothesis of no causality running from speculation to permanent shocks cannot be rejected in 7 out of the 9 commodities on conventional significance levels; there are two commodities with p values of 4.5 and 6% where the null hypothesis is rejected (Soybean and sugar). However, the estimated VAR coefficients²³ are negative for these two commodities, which imply a stabilizing price effect of speculation in the price adjustment process. The hypothesis of no Granger causality running from permanent shocks to speculation is rejected for a single commodity (cotton) with a p value slightly above 10%.

Table 4 Test for Granger causality between speculation and squared permanent as well as transitory shocks

The results of Granger causality between speculation and squared transitory shocks (TS) are displayed in the fourth and fifth column of Table 4. The null hypothesis can be rejected for only one commodity (heating oil); the p values for the other commodities are beyond conventional significance levels. Again, the estimated VAR coefficient is negative. There is no statistical evidence for the reverse relationship (transitory price effects causing speculation) for any of the commodities.

Summing up: While our results reveal a few permanent price effects of speculation, there is no general support for the hypothesis that speculation as measured by the Working T index is destabilizing commodity prices (as measured by squared innovations). In the very few significant cases, the effect from speculation is even stabilizing.

5.3 Variance decomposition

Based on the VAR models, a variance decomposition is performed for determining the fraction of variability in permanent and transitory shocks caused by speculation, and respectively the reverse relationships. The figures displayed in Table 5 are the largest variance proportions observed across the time lags; figures are marked in bold if the respective Granger causality tests reject a significant relationship on the 10% level.

Table 5 Variance decomposition: largest explained variance

The most interesting results are in column 2 which displays the fraction of permanent price shocks driven by speculation. The overall effects are small. The largest value can be found for crude oil with 8.2% (no Granger causality), followed by soybean (no Granger causality rejected) and heating oil (no Granger causality). The impact of speculative shocks on overall price variability is furthermore tempered by the observation from Table 3 that the largest portion of futures price variability is driven by transitory, not permanent shocks. The variance proportions in column 4 reveal that the impact of speculation on transitory price shocks is largest for heating oil, 5.3% (no Granger causality rejected), but is otherwise in the same order of magnitude as for the permanent shocks.

For completeness, the variance proportions of the shocks in the reverse direction (permanent and transitory shocks on speculation) are also documented, but need no further comment.

We conclude from these results that, overall, the variance of permanent and transitory shocks is driven by factors other than speculation; its impact is less than a few percentages.

6 Summary

This paper takes an innovative look at the highly controversial debate on commodity futures prices and speculation. Contrary to other studies on that subject, we disentangle transitory and permanent futures price shocks in addressing this issue. This distinction is important: Does speculation shift the long-run price level of commodities, or does it distort the price discovery process toward the equilibrium?

We analyze transitory and permanent futures price shocks from the VECM of a cointegrated system of pairwise short- and long-dated contracts for a sample of fourteen commodities. To minimize estimation problems, we just include two futures maturities in our system, the nearby contract and a “long”-dated contract expiring 12 months ahead of the nearby contract. The long-dated contract is not really long, but it is the longest common maturity across the selected commodities with price data covering two decades.

We find that the short- and long-dated futures price series of 9 commodities are cointegrated, i.e., establish a long-term price equilibrium. The VECM reveals that except for one commodity, cotton, the long-term equilibrium is determined by the long-dated contract, while the adjustment toward equilibrium is restored by the short-dated contract. The transitory shocks account for 65–100% of the variance of the short-dated contract (with typical values in excess of 90%), implying half lives between one and two years.

Granger causality tests cannot reject the null hypothesis that speculation as measured by Working’s T index has no effect on squared permanent (respectively transitory) price shocks for 7 (respectively 8) out of 9 commodities. Where the null hypothesis is rejected, the relationship exhibits a negative sign, i.e., speculation has a stabilizing effect. Therefore, there is no general support for the hypothesis that speculation is destabilizing commodity prices.

Appendix A: Data sources and construction

1.1 A.1 Futures Prices

Monthly futures price series are constructed in the following way: The short-dated futures prices reflect a rolling futures strategy with the shortest available contract²⁴. The long-dated futures price series is constructed analogously, but uses contracts expiring one year ahead of the nearby maturity. The contracts are rolled into the next available maturity in the month where the shortest contract expires; a fixed business day is selected for the rollover. The roll schedule applied to each commodity is available upon request. The selected time spread of exactly one-year controls for seasonalities in the term structure of commodity futures prices; i.e., by rolling the contracts, seasonal price “jumps” vanish because the contracts exhibit the same expiration month. All prices are denoted in US dollars and were downloaded from Thomson Reuters Datastream.

1.2 A.2 Speculation

The data used to calculate the T index are provided by the Commitments of Traders (COT) report released by the U.S. Commodity Futures Trading Commission (CFTC). It contains the allocation of open interest on each Thursday for markets in which 20 or more traders hold positions equal to or above the reporting levels established by the CFTC. Two main classifications are released: the Commitments of Traders Futures-Only Report contains futures market open interest only. Historical data files are available from 1986. Since March 14, 1995 the Futures-and-Options-Combined Report provides an aggregation of futures market open interest and delta-weighted option market open interest. The published open interest for each market is aggregated across all contract maturities in both reports. All data are released on Friday at 3:30 p.m. However, only mid-month and month-end data were provided by the COT before September 30, 1992. We combine the data from the two reports to construct a single time series for each commodity.

Both reports classify the positions into commercials, non-commercials, and non-reported. For each group, the respective number of long and short contracts is reported separately.²⁵ The aggregate of all long and short positions add up to the market’s total open interest.

Following common practice in the empirical literature, “commercials” are considered as hedgers, whereas “non commercials” are classified as speculators. However, the group of non-reporting traders cannot be easily classified as hedgers or speculators without strong assumptions. Sanders et al. (2010) point out that the speculation index is not particularly sensitive to the assignment of the non-reporting traders. For that reason, this group is omitted in computing our T index.