1 Introduction

Since the increase in commodity prices in the mid-2000s, some commentators, policymakers, and nongovernmental organizations (NGOs) have raised concerns that trading behavior of speculators started to predominate price levels in commodity markets. Commodity futures and options markets began to grow rather rapidly around 2004, both in held positions and traded volume, which is referred to as the ‘financialization’ of commodity markets (Irwin and Sanders 2012b). Even though only little evidence for direct influences on price levels from speculative positions and index funds has been found (see Irwin and Sanders 2011; Will et al. 2012; Gilbert and Pfuderer 2014), the public opinion in many countries is still dominated by the assumption that commodity futures prices are characterized by speculative pressures driven by futures market participants.Footnote 1 In this context, speculative pressures occur when (net) demand for long speculation exceeds (net) short hedging needs, which may result in changed price levels.

On the other hand, a number of trading guides present trading strategies that make use of the concept of hedging pressure to capture the risk premium in commodity futures markets (e.g., Upperman 2006; Briese 2008; Miffre 2012). Hedging pressure, based on the theory of ‘normal backwardation’ of Keynes (1930) and Hicks (1939), tries to explain the price behavior of futures in relation to hedgers’ position data. It is hypothesized that if the (net) demand for short hedging exceeds the demand for (net) long speculation, then long speculators will need to be compensated by an additional return risk premium to encourage them to balance the excess demand for short hedging, and this may result in price impacts.

Both concepts of hedging and speculative pressure assume explicitly or implicitly that movements in futures prices are directly affected by changes in hedgers’ or speculators’ open interest positions.Footnote 2 There have been a number studies on these relationships among traders’ positions and prices in commodity futures markets. Based on position data of hedgers or speculators, some previous studies have found some evidence for hedging pressure (e.g., Bessembinder 1992; Roon et al. 2000; Basu and Miffre 2013)Footnote 3 or speculative pressure (Cooke and Robles 2009; Gilbert 2010a, b; Plastina 2010; Singleton 2011). In contrast, other studies (e.g., Wang 2003; Bryant et al. 2006; Sanders et al. 2009)Footnote 4 have failed to find any evidence that either hedgers’ or speculators’ positions lead prices.Footnote 5 Nevertheless, the majority of studies report contemporaneous relationships between position data of market participants and prices. These observed contemporaneous relationships may be interpreted as pressure from positions to prices; however, correlations between price changes and net position changes of market participants cannot indicate any causation, and causal linkages are theoretically not obvious.

This article contributes with a comprehensive and systematic empirical investigation on the lead–lag relationships among traders’ positions and prices over a broad range of commodity markets (agricultural, livestock, softs, energy, and metals). In contrast to previous studies (e.g., Sanders et al. 2004, 2009), an augmented Granger-causality test framework (Toda and Yamamoto 1995) that is robust even if time series variables are nonstationary (or cointegrated) is applied on a variety of measures of hedging and speculative activities. In addition to Granger-causality, tests on the cumulative directional impacts between prices and position variables are conducted (Sanders et al. 2009). Recent futures position data considered include weekly issued data from Commodity Futures Trading Commission (CFTC) Commitments of Traders (COT), 1995–2013, and Commodity Index Trader Supplement (CIT) reports, 2006–2013.Footnote 6 Furthermore, possible structural changes in the relationships among traders’ positions and prices in futures markets are analyzed by examining two subsamples (1995–2003, 2004– 2013) for COT report data.Footnote 7 The analysis in this paper shows whether findings of previous research are robust to different measures of hedging and speculative activities, different position data categories, extensions over time, and markets and alternative test frameworks. In this context, it contributes to the literature with conclusive and robust empirical evidence on the effects of hedging and speculative pressures.

2 Data and methods

2.1 Commitments of traders and commodity index trader supplement reports

Futures positions of market participants are publicly available from CFTC’s reports on a weekly basis. The futures market open interest positions of market participants are collected every Tuesday (aggregated across all contract expiration months for a given commodity) and made available to the public the following Friday at 3:30 p.m. EST. In this study, futures price data and position data of the COT reports are collected for eight agricultural (CBOT corn, oats, soybeans, soybean meal, soybean oil, and wheat; KCBT wheat; MGE wheat), three livestock (CME feeder cattle, live cattle, lean/live hogs),Footnote 8 five softs (ICE cocoa, coffee, cotton, orange juice, and sugar), four energy (NYMEX crude oil, heating oil, natural gas, and gasoline), and four metal (COMEX copper, gold, and silver; NYMEX platinum) futures markets over 19 years from March 1995 through December 2013.Footnote 9

In COT reports, the CFTC classifies traders based on the size of their positions into reportable and nonreportable (reporting traders hold positions in excess of CFTC reporting levels). Reportable traders constitute the majority of the open interest of any futures market and are further classified as commercial (hedgers) or noncommercial (speculators) traders. A trader’s futures position is determined to be commercial if the position is used for hedging purposes as defined by CFTC regulations. Futures positions are otherwise classified as noncommercial. The market’s total open interest (TOI) is disaggregated in the following way:

$$\begin{aligned} \underbrace{\overbrace{[\text {NCL}+\text {NCS} +2\cdot \text {NCSP}]}^{\text {Noncommercial}} +\overbrace{[\text {CL}+\text {CS}]}^{\text {Commercial}}}_{\text {Reporting}} +\underbrace{[\text {NRL}+\text {NRS}]}_{\text {Nonreporting}} =2\cdot \text {TOI}, \end{aligned}$$
(1)

with NCL, NCS, and NCSP are noncommercial long, short, and spreading positions, respectively.Footnote 10 CL (CS) represents commercial long (short) positions, and NRL (NRS) represents long (short) positions held by nonreporting traders. Reporting and nonreporting positions must sum to the market’s TOI, and the total number of long positions must be equal to the total number of short positions.

However, while the classification of commercial traders as hedgers and noncommercial traders as speculators may have never been fully accurate (e.g., Ederington and Lee 2002; Sanders et al. 2004),Footnote 11 the experienced structural changes in commodity futures markets and the increased diversity of futures market participants since the mid-2000s may have changed the composition of traders classified as commericals (e.g., Gilbert 2010b; Buyuksahin and Harris 2011; Irwin and Sanders 2012a). In particular, (mostly) long-only commodity index funds may be classified as commercials since their positions are hedged by swap dealers on the futures market (CFTC 2006). The swap dealers’ underlying risk may not be a position in the physical commodity market; rather, the reporting commercials may be financial institutions that hedge their risk of over-the-counter (OTC) derivative positions. Hence, there is uncertainty whether an underlying swap dealer position represents hedging or speculating behavior.

To address the issues of shortcomings in COT classifications, the CFTC started to publish more disaggregated reports. The Commodity Index Trader Supplement report (CIT), available from January 2006 to December 2013 for twelve agricultural futures markets,Footnote 12 adds the new category of index traders to commercials and noncommercial positions (both less index traders).Footnote 13 In CIT reports, the market’s total open interest (TOI) is disaggregated as follows:

$$\begin{aligned}&\underbrace{\overbrace{[\text {NCL}^{\text {-CITL}}+ \text {NCS}^{\text {-CITS}}+2\cdot \text {NCSP}]}^{\text {Noncommercial}} +\overbrace{[\text {CL}^{\text {-CITL}}+ \text {CS}^{\text {-CITS}}]}^{\text {Commercial}} +\overbrace{[\text {CITL}+\text {CITS}]}^{\text {CIT}}}_{\text {Reporting}}\\ \nonumber&+\underbrace{[\text {NRL}+\text {NRS}]} _{\text {Nonreporting}}=2\cdot \text {TOI}, \end{aligned}$$
(2)

with \(\hbox {NCL}^{\text {-CITL}}\) \((\hbox {NCS}^{\text {-CITS}}\)) and \(\hbox {CL}^{\text {-CITL}}\) \((\hbox {CS}^{\text {-CITS}}\)) are noncommercial long (short) and commercial long (short) positions less index trader long (short) positions. CITL (CITS) represents index trader long (short) positions, and NCSP, NRL, and NRS denote noncommercial spreading positions, nonreporting long, and nonreporting short, respectively. As for COT reports, reporting and nonreporting positions must sum to the market’s TOI, and the total number of long positions must be equal to the total number of short positions.Footnote 14

Following the literature (e.g., Wang 2003; Röthig and Chiarella 2007; Sanders et al. 2009), futures prices on Tuesdays’ closing are matched with COT/CIT report releases, which reflect trader’s positions as of Tuesdays’ close. The nearby futures contract is used by the nearest contract up to one month before maturity and then rolling their position to the second nearest contract.Footnote 15

2.2 Measurements of hedging and speculative positions and activities

A variety of measurements of position and pressure variables based on CFTC’s COT (1995–2013, 24 commodity markets) and CIT (2006–2013, 12 commodity markets) report data are used to test for lead–lag relationships between hedging and speculative activities and prices. Proxies for hedging and speculative pressures are calculated following the literature (e.g., Roon et al. 2000; Sanders et al. 2004) for COT and CIT report data, respectively. The percent net long position held by commercials is referred to as hedging pressure (HP) variable,

$$\begin{aligned} \text {HP}_t=\frac{\text {CL}_t-\text {CS}_t}{\text {CL}_t+ \text {CS}_t}\text {, and HP}_t^{\text {-CIT}}=\frac{\text {CL}_t^{\text {-CITL}} -\text {CS}_t^{\text {-CITS}}}{\text {CL}_t^{\text {-CITL}} +\text {CS}_t^{\text {-CITS}}}, \end{aligned}$$
(3)

the percent net long position held by noncommercials as speculative pressure (SP) variable,

$$\begin{aligned}&\text {SP}_t=\frac{\text {NCL}_t-\text {NCS}_t}{\text {NCL}_t +\text {NCS}_t+2\cdot \text {NCSP}_t}\text {, and}\end{aligned}$$
(4)
$$\begin{aligned}&\text {SP}_t^{\text {-CIT}}=\frac{\text {NCL}_t^{\text {-CITL}} -\text {NCS}_t^{\text {-CITS}}}{\text {NCL}_t^{\text {-CITL}} +\text {NCS}_t^{\text {-CITS}}+2\cdot \text {NCSP}_t}, \end{aligned}$$
(5)

and the percent net long position held by nonreporting traders as small trader pressure (STP) variable,

$$\begin{aligned} \text {STP}_t=\frac{\text {NRL}_t-\text {NRS}_t}{\text {NRL}_t+\text {NRS}_t}. \end{aligned}$$
(6)

Based on CIT report data, the percent net long position held by commodity index traders is referred to as commodity index trader pressure (CITP) variable:

$$\begin{aligned} \text {CITP}_t=\frac{\text {CITL}_t-\text {CITS}_t}{\text {CITL}_t+\text {CITS}_t}. \end{aligned}$$
(7)

Hedging, speculative, small trader, and commodity index trader pressure variables are bound to be between \(-\)1 and 1, and they are to be interpreted as follows. For example, a HP of \(-\)0.3 means that 30 % of commercials are net short. Vice versa, a SP of 0.3 means that 30 % of noncommericals are net long. The pressure variables HP, SP, and STP (and CITP for CIT report data), weighted by their percent of TOI, will sum to zero.

Moreover, Working’s (1960) ‘T’ speculative index is included as an additional proxy for speculative activity. For CIT report data, the index is adjusted by classifying index traders as speculators (e.g., Sanders et al. 2010):

$$\begin{aligned}&\text {T}_t={\left\{ \begin{array}{ll} 1+\frac{\text {NCS}_t}{\text {CL}_t+\text {CS}_t}, &{} \text {if }\text {CS}_t\ge \text {CL}_t, \text { or} \\ 1+\frac{\text {NCL}_t}{\text {CL}_t+\text {CS}_t}, &{} \text {if }\text {CL}_t>\text {CS}_t. \end{array}\right. }, \\ \nonumber&\text {T}_t^{\text {CITadj}}={\left\{ \begin{array}{ll} 1+\frac{\text {NCS}_t^{\text {-CITS}}+\text {CITS}}{\text {CL}_t^{\text {-CITL}}+\text {CS}_t^{\text {-CITS}}}, &{} \text {if }\text {CS}_t^{\text {-CITS}}\ge \text {CL}_t^{\text {-CITL}}, \text { or} \\ 1+\frac{\text {NCL}_t^{\text {-CITL}}+\text {CITL}}{\text {CL}_t^{\text {-CITL}}+\text {CS}_t^{\text {-CITS}}}, &{} \text {if }\text {CL}_t^{\text {-CITL}}>\text {CS}_t^{\text {-CITS}}. \end{array}\right. } \end{aligned}$$
(8)

This speculative index was proposed by Working (1960) as a measure of technically ‘excess’ speculation. Long and short hedgers will not always trade at the same time or in the same quantity, and speculators meet hedging demand. The T index has a minimum value of 1.00, when the level of speculation equals hedging needs. ‘Excess’ speculation, in this technical sense, means the level of speculation (noncommercial positions) relative to hedging (commercial positions). For example, an index level of 1.30 means that there is 30 % speculation in excess of that what is necessary to meet hedging needs. Working (1960) discussed that excess speculation is necessary for the functioning of futures markets, and the majority of prior studies (e.g., Labys and Granger 1970; Peck 1980; Leuthold 1983) were concerned about the lack of speculation to meet hedging needs.Footnote 16

2.3 Augmented Granger-causality test

To test for lead–lag relationships among position variables and prices, bivariate Granger-causality tests are conducted in a vector autoregression (VAR) framework (Lütkepohl 2007). The concept of Granger-causality can be explained as follows. In the case of two time series, \(\hbox {Ps}_t\) and \(\hbox {Pr}_t\), \(\hbox {Ps}_t\) Granger-causes \(\hbox {Pr}_t\) if \(\hbox {Pr}_t\) can be better predicted using the histories of both \(\hbox {Ps}_t\) and \(\hbox {Pr}_t\) than it can by using the histories of \(\hbox {Pr}_t\) alone.

However, Granger-causality test statistics have nonstandard asymptotic properties if the VAR contains nonstationary (and possibly cointegrated) time series variables, and thus, conventional tests are not valid (Toda and Phillips 1993; Lütkepohl 2007, Ch. 7). To avoid this problem, the majority of previous studies on relationships between positions and prices (e.g., Röthig and Chiarella 2007; Sanders et al. 2009; Gilbert 2010a, b) have simply differenced possibly nonstationary time series data, and they do not tend to consider possible cointegration.Footnote 17

In this study, an augmented Granger-causality framework proposed by Toda and Yamamoto (1995) is used that is robust to integrated or cointegrated time series contained in VAR models. The Toda–Yamamoto Granger-causality framework is as follows. First, the maximum order of integration \(d_{max}\) of the variables considered has to be determined, e.g., by Augmented Dickey–Fuller tests (ADF). Then, the optimal lag order \(p\) of the VAR model has to be selected using multivariate Information Criteria, e.g., by Bayesian information criterion (BIC). Next, the lag-augmented VAR\((p+d_{max}\))

$$\begin{aligned} \begin{bmatrix} \text {Pr}_{t}\\ \text {Ps}_{t} \end{bmatrix} =\sum _{i=1}^{p+d_{max}} \begin{bmatrix} \gamma _{1,i}&\gamma _{2,i}\\ \gamma _{3,i}&\gamma _{4,i} \end{bmatrix} \begin{bmatrix} \text {Pr}_{t-i}\\ \text {Ps}_{t-i} \end{bmatrix}+ \begin{bmatrix} \alpha _{1}\\ \alpha _{2} \end{bmatrix}+t \begin{bmatrix} \beta _{1}\\ \beta _{2} \end{bmatrix} +\begin{bmatrix} \varepsilon _{1,t}\\ \varepsilon _{2,t} \end{bmatrix} \end{aligned}$$
(9)

is estimated, with \(\hbox {Pr}_{t}/\hbox {Ps}_{t}\) denoting price/position variables. In this study, the VAR in Eq. 9 is modeled with additional constant terms \(\alpha _{1}\) and \(\alpha _{2}\), linear time trend terms \(\beta _{1}\) and \(\beta _{2}\), and \(\varepsilon _{1,t}\) and \(\varepsilon _{2,t}\) are error terms. \(\hbox {Ps}_t\) is not Granger-causal for \(\hbox {Pr}_t\) if the bivariate VAR\((p+d_{max})\) process has \(\gamma _{2,i}=0\), for all \(i=1,2,...,p\). That is, it requires checking whether specific coefficients are jointly zero (by an \(F\) test), and testing \(\hbox {H}_0\): \(\gamma _{2,i} = 0, \forall i\le p\) is a test that \(\hbox {Ps}_t\) does not Granger-cause \(\hbox {Pr}_{t}\). Similarly, testing \(\hbox {H}_0\): \(\gamma _{3,i} = 0, \forall i\le p\) is a test that \(\hbox {Pr}_{t}\) does not Granger-cause \(\hbox {Ps}_{t}\). For both cases, a rejection of the null hypothesis implies that there is Granger-causality. Note that the Granger-causality null hypothesis of zero coefficients is tested on only the first \(p\) coefficients.Footnote 18

2.4 Cumulative directional impact test

In addition to Granger-causality, tests on the cumulative directional impact from one time series variable to the other are of interest (e.g., Sanders et al. 2009). For example, the cumulative impact from \(\hbox {Pr}_{t}\) to \(\hbox {Ps}_{t}\) is tested by the null hypothesis (\(F\) test distribution)

$$\begin{aligned} \text {H}_0:\text { } \sum _{i=1}^p \gamma _{3,i} = 0. \end{aligned}$$
(10)

If \(\gamma _{3,i} = 0, \forall i\le p\) and \(\sum _{i=1}^p \gamma _{3,i} = 0\) are rejected (i.e., there is Granger-causality and a directional impact), then there is ‘positive feedback’ from \(\hbox {Pr}_{t}\) to \(\hbox {Ps}_{t}\) when \(\sum _{i=1}^p \gamma _{3,i} > 0\), or ‘negative feedback’ from \(\hbox {Pr}_{t}\) to \(\hbox {Ps}_{t}\) when \(\sum _{i=1}^p \gamma _{3,i} < 0\). In the context of traders’ positions and prices, a ‘positive feedback’ indicates trend followers since traders tend to increase their long position after prices increase, and vice versa. On the other hand, a ‘negative feedback’ indicates contrarians since traders tend to buy after price declines and sell after price increases.

3 Empirical results and discussion

3.1 Summary statistics and contemporaneous correlations

Commodity futures markets have indeed experienced a tremendous increase both in market size and prices since the mid-2000s (e.g., Stoll and Whaley 2010; Buyuksahin and Harris 2011; Irwin and Sanders 2012b). For example, Fig. 1 depicts weekly total open interest (COT reports) and nearby futures prices of selected commodity markets, March 1995 through December 2013. By visual inspection, patterns of positions and prices may easily suggest a tendency of the variables to increase in tandem since the mid-2000s. Summary statisticsFootnote 19 of prices and variables for COT and CIT report data used in this study imply a wide variation of positions and prices, increasing more than tenfold for some commodities from minimum to maximum. ADF-GLS test results suggest that prices tend to be I(1), i.e., order of integration of one, while mixed evidence of I(0) and I(1) is found for position and pressure variables.Footnote 20 The variables for hedging and speculative pressures have negative and positive mean values, respectively. That means, on average, hedgers are net short while speculators are net long. However, standard deviations, minimum, and maximum values indicate substantial variation across commodities and time. The mean values for small trader pressure variables suggest a somewhat mixed pattern with some positive and some negative values, indicating a range of different hedging or speculative behavior of small nonreporting traders. Since commodity index funds are mostly long-only, index trader pressure variables are always positive, meaning that index trader are net long.

Fig. 1
figure 1

Weekly total open interest and nearby futures prices of corn, live cattle, crude oil, and gold, March 1995 through December 2013

Full size image

Contemporaneous correlations between changes in prices (returns) and positions are highly significant and consistent in signs across commodities and COT/CIT report data for position variables for hedgers and speculators, and hedging and speculative pressure variables. Changes in hedging and speculative pressures are negatively and positively associated with returns, respectively. Price returns appear to have a positive relationship with changes in speculative long and hedging short positions. On the other hand, returns show a negative relationship with speculative short and hedging long position changes. For nonreporting positions and small trader pressure, the results are mixed, indicating again the heterogenous behavior of nonreporting traders. In addition, commodity index trader positions show less consistent results, with only index trader long position variables mostly significant and always positively related to price returns across commodities. Finally, changes in the speculative T index tend to have a negative relationship with price returns.

Note again, however, that contemporaneous relationships do not indicate that either long speculative, short hedging, and long index trader positions, or speculative pressures lead to increasing prices, as they do not indicate that either short speculative and long hedging positions, hedging pressures, or the speculative index lead to decreasing prices. In principle, the contemporaneous relationships could reflect four different systematic lead–lag relationships: Positions may lead prices as assumed by the concepts of hedging and speculative pressures, prices may lead positions as previous prices are an input of trading behavior, the lead–lag relationships may be bidirectional, or there is no lead–lag relationship at all and prices and positions react contemporaneously to another factor.

3.2 Are prices led by position activities of groups of market participants?

Table 1 shows test results for Granger-causality and cumulative directional impacts from position variables to prices, for COT and CIT report data, estimated over the full sample periods (1995–2013, and 2006–2013) and all commodities (24 and twelve commodity futures markets).Footnote 21 The reported figures denote the cumulative directional impact (i.e., \(\sum _{i=1}^p \gamma _{2,i}\) to measure and test impacts from \(\hbox {Ps}_{t}\) to \(\hbox {Pr}_{t}\)), and asterisks and daggers indicate significance for Granger-causality and cumulative directional impacts tests, respectively. For the vast majority of commodities and position variables, the null hypotheses of no Granger-causality and no directional impacts cannot be rejected. In particular, for pressure and speculative index variables, the null hypothesis that pressure variables do not lead prices could only be rejected for a few commodities based on COT reports (six for hedging pressure, three for speculative pressure, and five for the speculative index) and CIT reports (two for hedging pressure and one for index trader pressure, speculative pressure, and the speculative index, respectively). In addition, the signs of directional impacts do not indicate any consistent direction across commodity markets. There is, therefore, hardly any indication that positions systematically lead prices across commodities.

Table 1 Toda–Yamamoto Granger-causality and cumulative directional impact test results, positions to prices, COT (1995–2013) and CIT reports (2006–2013)
Full size table

While over the full sample period of COT report data (1995–2013) test results for Granger-causality and cumulative directional impacts suggest little relationships, structural changes experienced in commodity futures markets may have affected the relationships among trading positions and prices.Footnote 22 For COT report data, test results for lead–lag relationships from trading position variables to prices in subsample periods (1995–2003, 2004–2013) in Table 2 do not indicate any substantial change. Consistent with the full sample, for the majority of commodities and position variables, the null hypotheses of no Granger-causality and no directional impact cannot be rejected. For example, the null hypothesis of no Granger-causality is rejected only for four and three commodities for hedging and speculative pressure in the first period and only for three and two commodities for hedging and speculative pressure in the second period, respectively. As for the entire sample period, signs of directional impacts do not suggest any consistent direction across commodity markets and periods. That is, there is little indication that structural changes in commodity futures markets have changed lead–lag relationships from position variables to prices, and positions still do not appear to lead prices in commodity futures markets.

Table 2 Toda–Yamamoto Granger-causality and cumulative directional impact test results, positions to prices, COT reports, subsample periods (1995–2003, 2004–2013)
Full size table

3.3 Are positions of groups of market participants led by prices?

In contrast to relationships from position variables to prices, Granger-causality test results in Table 3 do indicate systematic lead–lag relationships from prices to position variables over the full sample and almost all commodities. In particular for COT report data, speculative long/short, hedging short, the pressure variables, and the speculative index appear to be systematically led by prices. Most important, the null hypothesis for no Granger-causality is rejected for all 24 commodities for speculative pressure and 22 for hedging pressure. Out of the 24 commodities tested, Granger-causality is found for small trader pressure for 19 commodities, for the speculative index for 21 commodities, hedging short 18 commodities, and speculative long/short for 18 and 23 commodities, respectively.

Table 3 Toda–Yamamoto Granger-causality and cumulative directional impact test results, prices to positions, COT (1995–2013) and CIT reports (2006–2013)
Full size table

Cumulative directional impacts tend to show a consistent pattern for some position variables. Since significant directional impact measures are negative and positive for eleven and six tested commodities for speculative short and nonreporting long, respectively, speculators tend to decrease their short positions and nonreporting traders tend to increase their long positions after prices increase. In addition, results for directional impact measures for speculative and small trader pressure show ‘positive feedbacks’ since significant and positive results could be obtained for seven commodities for speculative pressure and eleven commodities for small trader pressure (however, two significant results are negative). This may indicate that speculators and small traders tend to be trend followers. On the other hand, hedgers tend to buy after price declines and sell after price increases since all eight significant test results for hedging pressure show negative directional impacts. This may imply that hedgers are contrarian traders.Footnote 23 Finally, the speculative T index tends to show ‘negative feedback’ since all 18 significant test results in the directional impact are negative. The results of the directional impacts from prices to position variables appear to correspond to contemporaneous correlations. Thus, these findings indicate that the contemporaneous relationship between prices and positions may be explained by directional impacts from prices to positions, and not vice versa.

While directional impacts based on CIT report data tend to be consistent in signs with results obtained based on COT, however, lead–lag relationships are less significant across commodities. Out of twelve agricultural commodities included in CIT reports, Granger-causality is found for speculative and hedging short for ten and five commodities, for speculative pressure for seven, for the speculative index for eight, and for hedging pressure only for four commodities. Interestingly, since for index trader long/short and index trader pressure Granger-causality and cumulative impacts can in each case only be significantly found for one commodity market and signs of impact measures do not tend to be consistent across commodities, commodity index traders do not appear to be led by previous prices.Footnote 24

To analyze whether structural changes in commodity futures markets have changed lead–lag relationships from prices to positions, Granger-causality and directional impact test results for the subsample periods based on COT report data are reported in Table 4. For the first period (1995–2003), out of 24 tested commodities, Granger-causality is found for 18 and all 24 commodities for hedging and speculative pressure, eleven for small trader pressure, 13 for the speculative index, 21 and 22 for speculative long/short, and eight and eleven for hedging long/short, respectively. For the second period (2004–2013), prices Granger-cause positions for 18 commodities for hedging and 21 for speculative pressure, 15 for small trader pressure, 18 for the speculative index, eleven and 20 for speculative long/short, and eight and 13 for hedging long/short, respectively. In addition, signs and test results of directional impacts lead to the same interpretations for both periods: Speculators tend to be trend followers (who buy after price increases and sell after price declines) and hedgers contrarian traders (who buy after price declines and sell after price increases). Overall, results strongly suggest that structural changes in commodity futures markets have not had substantial impacts on relationships between position variables and prices.

Table 4 Toda–Yamamoto Granger-causality and cumulative directional impact test results, prices to positions, COT reports, subsample periods (1995–2003, 2004–2013)
Full size table

3.4 Robustness analysis

Results of lead–lag relationships between position variables and prices may be sensitive to alternative position and price data, different subsample periods, and alternative Granger-causality test approaches. First, the COT open interest data used in this study are futures plus futures–equivalent options positions. Futures-only COT data are weekly available from the CFTC on a longer time period, from October 1992 to December 2013.Footnote 25 Thus, to examine whether results are sensitive to alternative COT data, all Granger-causality tests are also applied on futures-only data on the entire available period and two subsamples (October 1992–December 2003, January 2004–December 2013). Next, there is an inconsistency between COT data and futures prices. While COT data are aggregated across all contracts, only the nearby futures price is used. Although open interest tend to be largest in the nearby contract, futures prices for the first, second, and third deferred contracts, and open-interest-weighted futures prices based on the nearby and the three deferred contracts, are alternatively used in the tests.

Moreover, while futures markets and prices began to grow around 2004, changed relationships between positions and prices could have started to evolve later in time. Thus, tests are applied on alternative subsample periods (second period starting in 2005, 2006, 2007, and 2008). Then, to assess whether test results are sensitive to the Toda–Yamamoto framework, all Granger-causality and directional impact tests are also conducted on differenced time series data to avoid nonstationarity, i.e., on price return and changes of position variables, without lag augmentation.Footnote 26 All test results based on these alternative price and position data, different periods, and the alternative test approach lead to the same conclusions: systematic lead–lag relationships from prices to position variables, but not vice versa, and results are generally consistent between subsample periods.Footnote 27

3.5 Discussions and interpretations of results

The presented results in this paper provide a statistically rather conclusive picture; however, they may be subject to some discussion in their interpretations. For example, rejected Granger-causality tests should not be viewed as true ‘cause’ and ‘effect’ relations but only be interpreted as lead and lag relations between prices and hedging and speculative activities (e.g., Hamilton 1994; Gilbert 2010b; Buyuksahin and Harris 2011). Economic agents’ expectations on the future could make it appear as if there is a relationship between two variables when in fact there is none, or the observed lead–lag relationship may actually be the result of a third omitted variable (Lütkepohl 1982). It is possible that prices and trading activities react to the same common factors without any ‘causal’ relationship.

Another limitation of Granger-causality tests is that they are not fully capable of explaining the contemporaneous correlation. Granger-causality tests will capture these relationships only in so far as previous values of position variables are helpful in forecasting current prices, and vice versa (Hamilton 1994). Thus, the test power on hypotheses on relationships between hedgers’ and speculators’ positions and prices may be weak. The available weekly data may also mask relationships within weeks (e.g., Gilbert 2010b; Irwin and Sanders 2011, 2012a).

Finally, while ‘causal’ relationships could be much more dynamic (e.g., time-varying or nonlinear), Granger-causality tests will only be capable of detecting systematic patterns. That is, presented test results cannot rule out the possibility that pressures on prices from hedging or speculative behavior exist over specific or short intervals of time. Thus, it cannot be concluded that pressures from hedging or speculation do not exist at all. Only systematic lead–lag relationships from position variables to prices do not seem to be a characteristic of commodity futures markets, and this does not appear to have substantially changed given structural changes in markets.

Nevertheless, even if there were lead–lag relationships from pressures of hedgers and speculators on prices that are more dynamic, only within limited or short periods of time, or simply not detectable due to power properties of Granger-causality tests, lead–lag relationships from prices to hedging and speculative positions would still predominate. Granger-causality tests appear to have enough power to detect that position variables are generally affected by prices, and these relationships seem to be (at least) linear, systematic, and consistent over commodities and sample periods. Thus, even in the possible case of ‘true’ bidirectional causality, the ‘causal direction’ that prices tend to lead variables on hedging and speculative activity would still predominate, and hedging and speculative pressures may themselves be driven by feedbacks from previous prices.

4 Conclusions and implications

There has been a long and ardent debate whether pressures of hedging or speculation lead prices. Indeed, the contemporaneous relationship between price changes and (net) position changes of market participants is easily interpreted as pressure from positions to prices, but empirical evidence suggests rather little systematic lead–lag relationship. In this study, for the vast majority of commodities and position variables based on COT and CIT report data, the null hypothesis of no Granger-causality cannot be rejected. Furthermore, structural changes in trading and participants in commodity futures markets do not appear to have changed these relationships since results obtained for subsample periods are generally consistent. Thus, these empirical findings cast considerable doubt on systematic effects of hedging and speculative pressures in commodity futures markets. This does not imply that pressures from hedging or speculation do not exist at all; however, they have very probably not been a systematic characteristic of commodity futures markets, in the period neither from 1992 through 2003, nor from 2004 through 2013.

In contrast, there is strong evidence that prices tend to lead variables on positions and on hedging and speculative activity. Granger-causality test results indicate systematic lead–lag relationships from prices to position variables over the full sample and the vast majority of commodities. In particular, based on COT report data, the null hypothesis for no Granger-causality is rejected for 22 and all 24 commodities for hedging and speculative pressure variables. In addition, results of directional impacts from prices to position variables correspond to contemporaneous correlations. Speculators and nonreporting traders tend to be trend followers (who buy after price increases and sell after price declines) and hedgers contrarian traders (who buy after price declines and sell after price increases). These findings indicate that the contemporaneous relationship between prices and positions may be explained by directional impacts from prices to positions, and not vice versa. Furthermore, results based on CIT reports suggest that positions of commodity index traders, however, do not appear to react systematically to previous prices.

Importantly, these results generally hold across a range of different commodity futures markets (agricultural, livestock, softs, energy, and metals), over periods before and after the financialization of commodity markets, and different periods when market prices are stable, trend upward and are volatile. The results are also robust to a variety of measurements of hedging and speculative activities, COT and CIT report data, futures-only and futures plus delta-adjusted option data, prices from nearby and deferred futures contracts, and they are robust to different test frameworks.

In conclusion, there is little reason to assume that systematic effects of hedging and speculative pressures are helpful in explaining prices in commodity futures markets. To the contrary, prices may be helpful in explaining hedging and speculative position behavior. The findings of this paper contribute to the literature by providing empirical evidence that the financialization of commodity futures markets does not appear to have substantially changed systematic relationships among hedging and speculative activities and prices. Specifically, it supports and contributes with robust and conclusive evidence to findings of prior studies on previous time periods (e.g., Wang 2003; Bryant et al. 2006) and particular commodities in agricultural (e.g., Röthig and Chiarella 2007; Sanders et al. 2009) and energy futures markets (e.g., Sanders et al. 2004; Buyuksahin and Harris 2011) that neither hedging nor speculative positions may lead prices. This does still hold even after the emergence of new financial market participants and the start of the growth of commodity futures markets that began around 2004.

These findings are of importance for policymakers, regulators, and traders alike. The public policy debate may take note that neither hedging nor speculative positions appear to systematically lead prices, and consequently, policy decisions simply aimed at limiting speculative positions and trading in commodity futures markets are not supported by empirical evidence, and it is unclear whether they are beneficial for the functioning of price discovery and hedging in these markets. Indeed, results suggest that hedging and speculative positions follow prices, and not the other way round. Implications for traders are that, while conducted tests cannot rule out that position data may be useful in combination with other information, COT and CIT report data provide little direct insight to predict future prices. In fact, as traders’ positions respond to price changes, previous prices may be an input for hedging and speculative decisions.

While this study contributes to questions on systematic relationships between hedging, speculative, and index trader positions and prices in commodity futures markets, more issues remain. For example, further research is needed to examine the role of groups of traders in co-movements among different commodity and with equity markets (Tang and Xiong 2010; Buyuksahin and Robe 2011, 2012; Lehecka 2014). In addition, other questions include how relationships between prices and traders’ positions behave during times of price bubbles (Gilbert 2010b; Etienne et al. 2012, 2014; Gutierrez 2013).