The Behaviour of Sentiment-Induced Share Returns: Measurement When Fundamentals Are Observable

Abstract

We test the effect of sentiment on returns using a sample of upstream oil stocks where we have a good proxy for fundamental value. For this sample, the influence of sentiment is highly time-varying, appearing only after the post-2000 increased interest in oil-related assets. Contrary to the hard-to-arbitrage hypothesis, sentiment affects returns on these stocks principally through their fundamentals rather than through deviations from fundamentals. Retail investor sentiment predicts short-term momentum of fundamentals and Baker–Wurgler sentiment predicts mean reversion of fundamental factors. These effects appear in a portfolio that is long hard-to-arbitrage stocks and short easy-to-arbitrage stocks, but only because this portfolio has net exposure to fundamentals.

In this article we investigate the relationships between investor sentiment and deviations of share prices from fundamental values. To do this we use a sample of shares for which a large part of the fundamental value is observable: upstream oil stocks. We measure their fundamental values using oil and gas prices and the forward oil price contango.

We focus on upstream oil stocks because there is a direct relationship between the present value of these stocks and the oil price. In a world where output prices minus extraction costs obey the Hotelling Principle, the value of natural resource companies depends only on current prices less extraction costs. Miller and Upton (1985a, 1985b) test this proposition and find that it provides a good explanation of the variation in value of a sample of oil producers. Hence a large part of the fundamental value of upstream oil stocks is observable. We make use of this present value condition to split the return on our sample into the part that represents fundamentals and the part that is a deviation from fundamentals. The attraction of the Hotelling Principle is that it avoids the need to forecast future cash flows and to estimate discount rates. A more general and less parsimonious model might include additional variables, such as exchange rates and proxies for the discountrate.

Following Baker–Wurgler (2006), we test the impact of sentiment using a portfolio that is long high-variance stocks and short low-variance stocks (the “Hi-Lo” portfolio). We find that two types of sentiment predict returns: retail sentiment, which predicts momentum, and the Baker–Wurgler (2006) measure of sentiment, which predicts reversion to fundamentals. We find that the influence of sentiment in each case is time-varying. In particular, the ability of sentiment to predict returns appears only after 2000. Contrary to the theory that sentiment mainly affects the deviations from fundamental value of hard-to-arbitrage stocks, we find that both measures of sentiment influence prices through the fundamentals themselves rather than through deviations from fundamentals.

If the Hi and Lo portfolios had similar loadings on fundamental variables, the net Hi-Lo portfolio would be hedged against fundamental effects and we should not observe fundamentals affecting the returns on this portfolio. However, in our data the loadings on fundamental factors of the Hi and Lo portfolios are different, so the combination of a long and short position in this portfolio does not eliminate its exposure to fundamentals. Methodologically, this raises the issue that tests based on such portfolios do not avoid the need to control for fundamentals.

The remainder of the chapter is organised as follows. In Section 13.1 we give a brief review of related literature. In Sect. 13.2 we develop our tests and in Sect. 13.3 we describe our data. Section 13.4 presents our main tests of the influence of sentiment on returns with and without controls for fundamentals. Section 13.5 provides some robustness tests. Section 13.6 concludes.

13.1 Related Literature

Our work is broadly related to a number of studies that have found evidence of serial dependence in returns. Evidence of momentum over periods of six to twelve months is provided by amongst others Lehmann (1990), Jegadeesh (1990), Jegadeesh and Titman (1993, 2001), Asness, Moskowitz, and Pedersen (2013), and Moskowitz, Ooi, and Pedersen (2011). Evidence that this medium-term momentum is followed by longer-term mean reversion comes from variance-ratio tests (Poterba & Summers, 1988, Lo & MacKinlay, 1988, Cutler, Poterba, & Summers, 1991) and autocorrelation tests (Fama & French, 1988). Evidence that high short-term variance is related to deviations from fundamentals comes from excess variance tests (Shiller, 1981, LeRoy & Porter, 1981).

Sentiment-based explanations of momentum, mean-reversion, and deviations from fundamentals envisage these effects as arising from behavioural biases by naïve investors combined with costs of arbitrage. For example, Daniel, Hirshleifer, and Subrahmanyam (1998) present a model in which a combination of overconfidence and biased self-attribution create both under- and over-reaction. Similarly, Barberis, Shleifer, and Vishny (1998) appeal to the behavioural biases of representativeness and conservatism to show how these can result in under- and over-reaction. In both papers asset prices can be decomposed into one part that reflects fundamentals and another consisting of deviations from fundamentals. The effect of sentiment on asset prices operates through the deviations from fundamentals.

Empirical evidence on the link between sentiment and returns requires a measure of sentiment. A number of suggestions have been proposed. Many of these reflect the view that sentiment changes are driven by retail investors. Possible proxies include flows into mutual funds (Brown, Goetzmann, Hiraki, Shirishi, & Watanabe, 2003), buy–sell imbalances by retail investors (Kumar & Lee, 2006), IPO volume and initial returns (Baker & Wurgler, 2006), market turnover (Baker & Wurgler, 2006), closed-end fund discounts, (Lee, Shleifer, & Thaler, 1991, Chen, Kan, & Miller, 1993, Swaminathan, 1996, and Neal & Wheatley, 1998), the growth stock premium (Baker & Wurgler, 2006), and survey data (Qiu & Welch, 2004, Brown & Cliff, 2004, 2005). These data have been used either singly or in combination as sentiment measures to test hypotheses about the relationship between sentiment and subsequent stock returns.

Our tests of the effect of sentiment are most closely related to Baker and Wurgler (2006, 2007, 2012) and Baker, Wurgler, and Yuan (2012). Baker and Wurgler divide their sample of stocks into ten portfolios on the basis of their prior volatility, which serves as a proxy for difficulty of arbitrage. They find that returns on the more volatile stocks are lower following a time of optimism, and that returns are higher following a time of pessimism. For the less volatile stocks that are easier to arbitrage the reverse is true. They develop a measure of investor sentiment which they find predicts returns for portfolios that are long the more volatile stocks and short less volatile stocks. This finding is consistent with a combination of behavioural biases and limits to arbitrage.

Barberis, Shleifer and Wurgler (2005) stress the importance of controlling for fundamentals when measuring the effect of sentiment on security prices. For example, Derrien and Kecskés (2009) show that the effect of sentiment on equity issuance disappears once controls for fundamentals are included. Baker and Wurgler’s use of a Hi-Lo portfolio will be effective in controlling for fundamentals only if the long and short positions have equal loadings on fundamental factors. The alternative is to attempt to control directly for fundamentals. Brown and Cliff (2005) use as their dependent variable estimates of deviations from fundamental value based on the dividend discount model. They find that these deviations are positively related to a sentiment measure derived from survey data. However, the dividend discount model gives a very noisy observation of fundamental value. For our sample of stocks, we have a more direct measure of fundamental value than Brown and Cliff and so are able to perform a more powerful test of the way that sentiment is transmitted to share prices and returns.

13.2 Hypotheses and Tests

We test the implications of the hard-to-arbitrage hypothesis using a simple empirical procedure that relates sentiment measures to deviations of share prices from fundamentals and also to the fundamentals themselves. We split the log share price, P _t , into a component reflecting fundamental value, F _t , and a separate component, NF _t , which is the deviation from fundamental value:

$$ {\mathrm{NF}}_t={P}_t-{F}_t $$

(13.1)

We assume that prices are affected by the actions of two types of traders. One type is an arbitrageur, whose behaviour is captured by the Baker–Wurgler sentiment measure S _t . The other type is a naïve trend-follower, whose behaviour is captured by a bullish retail sentiment measure, B _t . Fundamentals and non-fundamentals may respond to both sentiment variables:

$$ {F}_t - {F}_{t-1}=\Delta {F}_t={\theta}_{B,F}{B}_{t-1}+{\theta}_{S,F}{S}_{t-1}+{\mathrm{e}}_{Ft} \vspace*{-8pt}$$

(13.2)

$$ {\mathrm{NF}}_t-{\mathrm{NF}}_{t-1}={\Delta \mathrm{N}\mathrm{F}}_t={\uptheta}_{B,\mathrm{N}\mathrm{F}}{B}_{t-1}+{\uptheta}_{S,\mathrm{N}\mathrm{F}}{S}_{t-1}+{\mathrm{e}}_{\mathrm{NF}t} $$

(13.3)

We model the response to Baker–Wurgler sentiment by assuming that arbitrageurs push prices down when this sentiment variable is high, making θ _S,F < 0 and θ _S,NF < 0. The effect of naïve trend-followers pushes prices up when Bullish sentiment is high, making θ _B,F  > 0 and θ _B,NF > 0.

The Baker–Wurgler sentiment variable is a measure of mispricing and so should rise when the deviation from fundamentals increases:

$$ {S}_t={\uptheta}_{S,S}{S}_{t-1}+{\uptheta}_{S,\mathrm{N}\mathrm{F}}{\Delta \mathrm{N}\mathrm{F}}_t+{\mathrm{e}}_{\mathrm{S}t} $$

(13.4)

where θ _S,NF > 0. We expect S _t to be highly persistent, reflecting the long cycle of swings in mispricing. We hypothesise that the bullish sentiment indicator reflects trend-following behaviour:

$$ {B}_t={\theta}_{B,B}{B}_{t-1}+{\theta}_{B,P}\left({P}_t - {P}_{t-1}\right)+{\mathrm{e}}_{Bt} $$

(13.5)

θ _B,P  > 0. We expect B _t to be less persistent than S _t , reflecting shorter cycles in momentum sentiment.

We stack Eqs. (13.2)–(13.5) to form the system shown below, which we estimate using VAR. Our null hypothesis is that the sentiment variables affect deviations from fundamentals but not the fundamentals themselves. This would be consistent with the limits-to-arbitrage hypothesis, whereby sentiment causes deviations from fundamentals when such deviations are hard to arbitrage. The indicated signs of the key parameters under the null hypothesis are shown in Table 13.1. In particular, the standard hypothesis is that sentiment affects deviations from fundamentals, implying θ _S,F  = 0, θ_B,F = 0, θ _S,NF < 0, θ _B,NF > 0. As an alternative, we test the hypothesis that the influence of sentiment occurs via its effect on fundamentals, which implies that θ _S,F  < 0, θ _B,F  > 0, θ _S,NF = 0, θ _B,NF = 0.

Table 13.1 Structure of the VAR and hypotheses about coefficients

13.3 Data

Our sample consists of the stocks of the 121 US oil exploration and production companies quoted on the NYSE during the period March 1983 to January 2011. We define an exploration or production company as one with a North American Industrial Classification (NAIC) of 211111 or a Standard Industrial Classification (SIC) of 1311. By limiting our sample in this way, we exclude refining companies that are likely to have very different loadings on the oil and gas factors.

We assume that fundamental value is a function of the month-end spot price of West Texas Intermediate oil and the spot wellhead price of West Texas natural gas. We also use the change in the contango in oil prices, where contango is measured as the log of the price of the sixth most distant futures contract less that of the price of the closest futures contract¹. The Appendix gives our data sources.

To investigate the behaviour of returns we form portfolios of oil stocks. We proxy the behaviour of all upstream oil stocks by the equally weighted portfolio of all stocks with return data for a given month (the “All-stocks” portfolio). The hard-to-arbitrage hypothesis suggests that sentiment-induced deviations from fundamental value should be greater in the more volatile stocks. We therefore form sub-portfolios of our sample of stocks consisting of the tercile of stocks with the highest variance of returns over the preceding 60 months and the tercile with the lowest variance. Following Baker–Wurgler, we form a portfolio that is long the tercile of high-variance stocks and short the tercile of low-variance stocks (the “Hi-Lo” portfolio). These portfolios are formed out of sample, and so represent a viable trading strategy.

We employ two measures of sentiment: the Baker–Wurgler index (“BW sentiment”) and the proportion of individual investors who report that they are bullish in the regular survey conducted by the American Association of Individual Investors (“Bullish sentiment”). Both measures are available for the period July 1987 to January 2011. To facilitate comparison between the two sentiment measures, we recalibrate the index values in terms of the number of standard deviations from the mean for the total period. Figure 13.1 provides a plot of these two rescaled measures. The Baker–Wurgler index is characterised by long swings in sentiment with a marked peak in value in February 2001. By contrast, Bullish sentiment is more noisy and less persistent. The first-order autocorrelation coefficient in the Baker–Wurgler index is .96, and the serial correlation in the index persists at least through lag 6. By contrast, the first-order autocorrelation coefficient in the AAII measure is .45 and the lower order serial correlations fall away rapidly.

Fig. 13.1

Measures of sentiment. This figure shows plots of the Baker–Wurgler and Bullish sentiment indexes from July 1987 to January 2011. In both cases the index has been standardised to have a mean of zero and standard deviation of unity

The monthly levels of the two sentiment indexes are only weakly related with a correlation of .09. The Baker–Wurgler index more closely resembles a cumulative sum of past values of the Bullish measure², which is consistent with the Baker–Wurgler index capturing cumulative deviation from fundamentals rather than short-term swings in sentiment.

Table 13.2 shows descriptive statistics for our variables. Panel A shows the means and standard deviations of the oil portfolio returns and of the changes in the fundamental values. Although the portfolios are formed out-of-sample, the standard deviation of the high-volatility portfolio is 50 % higher than that of the low-volatility portfolio. The volatility of the Hi-Lo portfolio which should, in principle, be hedged against changes in fundamentals is almost as high as that of the Hi volatility portfolio, suggesting that the long-short strategy may have only limited effect in controlling risk.

Table 13.2 Summary statistics

Panel B of Table 13.2 shows the correlation matrix for the entire period. Several features of the matrix are of interest and point to issues that are explored in more depth later.

• 1. The correlations between the returns on oil stocks and the three fundamental variables are quite high. Taken together, the three fundamental variables explain 41 % of the variance in the returns on the portfolio of all oil stocks.

• 2. The long-short portfolio of oil stocks (Hi-Lo) is not well hedged against fundamentals, and its returns remain quite highly correlated with all three fundamental variables. In other words, the high-volatility stocks are not only more difficult to arbitrage, but they also have different loadings on the fundamental factors.

• 3. There is little correlation between the fundamental variables and lagged sentiment. This suggests that controlling for fundamentals may not substantially change any estimate of the effect of sentiment, but also may make it easier to decompose portfolio returns into fundamental and sentiment components.

• 4. The correlations between returns and lagged sentiment are larger in absolute value for the high-variance portfolio and the long-short portfolio. This is consistent with the Baker–Wurgler cost-of-arbitrage hypothesis.

13.4 Sentiment and Returns

In this section we examine the influence of sentiment on returns. In Sect. 13.4.1 we provide evidence that the returns on the portfolios of oil stocks are characterised by momentum and longer-term mean reversion. We then examine the returns to see whether these patterns in returns are a function of our two measures of sentiment. In Sects. 13.4.2 and 13.4.3 we go on to decompose the returns into a fundamental and residual component and we analyse the relationship between these two components and our sentiment measures. In Sect. 13.4.4 we then examine the relationship between stock returns and “deep” fundamentals based on demand and supply in the oil market.

13.4.1 The Influence of Sentiment on the Hi-Lo Portfolio

Before testing for the effect of sentiment on returns, we first examine the serial properties of returns on the All-stock portfolio and the relationship of these returns to fundamentals. Table 13.3 shows for our portfolio of oil stocks the variance rates at differing intervals expressed as a proportion of the 1-month variance rate using the variance ratio test with overlapping data proposed by Lo and MacKinlay (1988). Consistent with standard results, the variance ratio rises for 6–9 months reflecting medium-term momentum and then falls back over the following year reflecting longer-term mean reversion.

Table 13.3 Variance ratios of the All-stocks portfolio

We examine the influence of sentiment by regressing total returns on the two lagged sentiment measures:

$$ {R}_t=a+{\theta}_{S,R}{S_{t-1}}_{+}+{\theta}_{B,R}{B}_{t-1}+{e}_{i,t} $$

(13.6)

To investigate the role of fundamentals we augment this regression with controls for the fundamental variables:

$$ {R}_t=a+{\theta}_{S,R}{S_{t-1}}_{+}+{\theta}_{B,R}{B}_{t-1}+{\theta}_{F,R}\Delta Ft+{e}_{i,t} $$

(13.7)

Where ΔFt is the vector of fundamental variables³.

Table 13.4 reports the result of regressions (13.6) and (13.7) for the All-stock portfolio and the Hi-Lo portfolio over the period 1988–2011⁴. For the All-stock portfolio before controlling for fundamentals the coefficient on the BW sentiment measure has the predicted negative sign but is insignificant, whilst the coefficient for the Bullish measure is positive and significant, as predicted. Including the fundamentals raises the R ² of the regression from .00 to .41. All three fundamental variables are highly significant with an unexpected 1 % increase in the oil price resulting in an increase of .59 % in the value of oil stocks. The coefficient on the change in the contango is positive suggesting that when the value of the future relative to the spot rises, there is a positive impact on the price of oil stocks. In other words, given the spot price of oil, the value of companies owning oil reserves increases when the futures price rises relative to the spot. Once these controls for fundamentals are included, the significance of the Bullish measure disappears and both sentiment measures become insignificant. The high R ² on the regression with fundamentals and the change in the significance of the sentiment measures shows the importance of controlling for fundamentals in testing the effect of sentiment.

Table 13.4 Regression of portfolio returns on lagged sentiment with and without controls for fundamentals

The hard-to-arbitrage hypothesis predicts that the influence of sentiment will be most marked for high-variance stocks. Therefore, following Baker–Wurgler, we also report in Table 13.4 the result of regressions (13.6) and (13.7) for the portfolio that is long high-variance stocks and short low-variance stocks. The high-variance stocks are characterised by a heavier loading on the three fundamental factors. This may stem from the higher relative importance of exploration (asopposed to production) activity as proxied by a low ratio of revenues to assets.⁵ Thus, despite its apparent hedged position, the Hi-Lo portfolio remains significantly exposed to oil and gas returns After controlling for fundamentals, the coefficients on both sentiment measures have the predicted sign, but only the Bullish variable is significant at the 10 % level.

For the Hi-Lo portfolio we also estimate these regressions for two sub-periods divided at 2000. The motivation for looking separately at the sub-periods is the sharp increase in institutional investment in oil futures after 2000 (Buyuksahin and Robe (2012) and Singleton (2011)). In the ten years to 2010 open interest in crude oil futures by non-commercial traders was 5.6 times its level over the previous 15 years (Fig. 13.2). Figure 13.3 shows that there was also a sharp rise in the cumulative cash flows into managed futures funds, which increased from $9.3 billion in September 2002 to $137.0 billion in March 2008 before losing most of these gains in 2009.

Fig. 13.2

Open interest in oil futures. This figure shows on a log scale the level of open interest in crude oil futures by non-commercial traders from January 1986 to December 2012. Source: CFTC

Fig. 13.3

Cumulative flows into commodity hedge funds. This figure shows on a log scale the cumulative quarterly cash flows into Managed Futures Funds in millions of dollars from first quarter 1994 to the third quarter 2010. Source: TASS Research

Table 13.5 repeats regressions (13.6) and (13.7) for the two sub-periods. There is a substantial difference between the sub-periods both in the coefficients and their significance. In the earlier period even before controlling for fundamentals neither of the sentiment coefficients is significant and the coefficient for the Baker–Wurgler measure has the wrong sign. In the second period, even after controlling for fundamentals, both sentiment measures have the predicted signs and remain significant at the 10 % level or better.

Table 13.5 Sub-period regressions of Hi-Lo portfolio returns on lagged sentiment with and without controls for fundamentals

13.4.2 Tests Using Fundamentals and Deviations from Fundamentals

We now test the hypothesis that sentiment operates by affecting deviations from fundamental value. To do so, we first decompose returns on the Hi-Lo portfolio into a fundamental component and a residual, and we then estimate the relationship between our sentiment measures and each of these two components. Sincetheloadings of this portfolio on the fundamental factors may be time-varying we estimate the loadings using rolling 60-month regressions of portfolio returns on the fundamental variables:

$$ {R}_{it} = {a}_i+{b}_i\Delta\ {\mathrm{WTI}}_t+{c}_i\Delta\ {\mathrm{Gas}}_t+{d}_i\Delta\ {\mathrm{Cont}}_t+{u}_{it} $$

(13.8)

We use the coefficients from this regression over the prior 60 months combined with the change in the month’s fundamentals to estimate that part of the month’s return that was due to fundamentals. The difference between the actual return in a month and the fundamental return is the residual, or non-fundamental, return.

We estimate the VAR system Eqs.(13.2)–(13.5) using GMM with the Newey–West correction for standard errors. Table 13.6 shows the results for the entire period 1988–2011⁶. Contrary to the “deviations from fundamentals” hypothesis all the coefficients of the regression of these deviations on lagged variables in column (2) are insignificant, and the Rbar² is negative. In contrast, the regression of fundamental returns on lagged sentiment (column (1)) has an Rbar² of 5 %. There is a negative coefficient on lagged B–W sentiment and a positive coefficient on the lagged bullish variable. Both coefficients are strongly significant. This result is consistent with sentiment-based trading operating largely through the fundamentals themselves, rather than through the deviations in the share price from fundamentals. The negative coefficient on lagged B-W sentiment is consistent with high sentiment signalling that the oil market is above its equilibrium and likely to fall. The positive coefficient on the Bullish variable is consistent with short-term momentum pushing the oil market upwards when retail investors are bullish.

Table 13.6 VAR of fundamental values for the Hi-Lo portfolio, deviations from fundamentals, Baker–Wurgler sentiment, and Bullish sentiment

Column (3) shows that the Baker–Wurgler sentiment variable is persistent, with a partial serial correlation of 0.97 consistent with a half-life of 20 months. The variable also responds to lagged changes in fundamentals, but not to lagged changes in deviations from fundamentals. Again, this is consistent with sentiment operating through the fundamentals themselves and not through deviations of shareprices from fundamentals. Column (4) shows that the Bullish variable is much less persistent, with a half-life of less than a month. It has positive serial correlation and positively responds to past fundamental returns.

Table 13.7 decomposes the Hi-Lo portfolio data into two sub-periods, divided at the end of 2000. Column (1) shows the results for the entire period, and columns (2) and (3) for the two sub-periods. The two sentiment variables have no significant effect on deviations from fundamentals in any period, though the coefficients on the Bullish variable consistently have the correct sign. The coefficients from the regression of the fundamental component of returns on the two lagged sentiment variables have the correct sign but are insignificant in the first sub-period. By contrast, in the second period the corresponding coefficients are strongly significant. Although the time-series behaviour of the sentiment variables appears to be the similar in the two sub-periods, the effect of sentiment on stock returns changes completely in the second period. Consistent with the result that sentiment affects prices largely via fundamentals, the effect occurs only once there is significant investment interest in the fundamental markets post-2000⁷.

Table 13.7 Sub-period VAR results

13.4.3 The Effect of the Differencing Interval

Table 13.8 shows the effect of increasing the differencing interval to 3 months and 12 months. The results are shown for the entire period (Panel A) and the two sub-periods (Panels B and C). The VAR is estimated using GMM with overlapping observations and Newey–West corrected standard errors. In the regression of fundamental returns the effect of moving from a 1-month to 3-month differencing interval is to increase the magnitude and significance on both of the lagged sentiment measures in all three periods. The Rbar² of the regression of fundamentals on sentiment increases dramatically, rising to .29 for a 12-month differencing interval in the second sub-period. Thus the sentiment measures appear to have a prolonged effect on the fundamental returns. In contrast, the longer differencing intervals have almost no effect on the coefficients for the deviations from fundamentals, which remain insignificant at all intervals and in all periods.

Table 13.8 Effect of sentiment on returns using different measurement intervals

13.4.4 Deep Fundamentals

Our measure of the fundamental return on the portfolio of oil stocks is equivalent to a weighted average of the contemporaneous change in the price of oil and gas and the change in the contango. The evidence that this weighted average is a function of the prior level of sentiment implies that oil and gas prices are themselves influenced by sentiment (Pindyck, 1993). Thus it appears that sentiment drives oil prices away from equilibrium values in a way that leads to predictable returns on oil stocks. This effect increases after 2000 when interest in commodities as an asset class increased significantly. 1-month returns are slightly predictable using sentiment, but returns over a 1-year horizon are highly predictable. Overall, the results appear to reflect a slowly changing but predictable component of oil and gas prices that is related to sentiment and generates a predictable return on oil stocks. Sentiment appears to have no effect on the price of oil stocks other than through the prices of the commodities themselves.

We can gain some further insight by examining the relationship between the change in oil prices and prior sentiment while controlling for the deeper fundamentals that determine oil prices.

$$ {\Delta \mathrm{WTI}}_t=a+{\theta}_S{S}_{t-1}+{\theta}_B{B}_{t-1}+{\theta}_F\Delta D{F}_t+{\mathrm{e}}_t $$

(13.9)

where ΔDF _t is the vector of the underlying determinants of the change in oil prices.

The main problem in estimating Eq. (13.9) is the lack of good proxies for deeper fundamentals that are available at sufficiently high frequency. We proxy the fundamental determinants oil prices by changes in world oil production and consumption, changes in world proven reserves (annual data only), changes in oil inventories (monthly data only), and a measure of economic growth. We estimate Eq. (13.9) using annual data, overlapping 12-month data and overlapping 3-month data. In the case of the annual data estimates are for the period 1988–2011 and in the case of the regressions using monthly data estimates are for the period 1994–2011. The results are summarised in Table 13.9.

Table 13.9 Regression of changes in the oil price on “deep” fundamental factors and lagged sentiment

The controls for fundamental variables have little explanatory power in the overlapping 3-month data regressions but in each case the coefficients on the Baker–Wurgler measure are negative and those on the Bullish measure are positive. However, with relatively few independent observations, the tests lack power and in only two cases is the coefficient on sentiment significant at the 10 % level. Thus the table provides mild direct support that sentiment affects energy prices and thereby the return on oil stocks.

13.5 Robustness Tests

We have already noted that our findings are robust to (a) pre-whitening the fundamental variables, (b) using different definitions of the crude oil price and the oil contango, (c) using the Datastream index of oil stocks.

13.5.1 Long-Only Portfolios

To the extent that the Hi-Lo portfolio is better hedged against fundamental factors than long-only portfolios, the fundamental component of returns will be relatively small. We therefore repeated the VAR estimates with long-only portfolios. The results for the tercile of stocks with the highest variance were very similar to those for the Hi-Lo portfolio. In particular, the effects of sentiment on returns were significant only for the second period, and sentiment impacted returns largely through the fundamental component.

13.5.2 Nasadaq Stocks

To test further the robustness of these results to a different sample of oil stocks, we extended the sample to include 274 stocks of US oil and exploration companies that were traded on Nasdaq. Although this produced a larger sample, the quality of the Nasdaq data appears to be inferior with shorter time series for many stocks, leptokurtic returns and more zero returns. Unsurprisingly, the high variance portfolio tended to have a high concentration in Nasdaq stocks.

Table 13.10 summarises estimates of regression Eq. (13.8) for the expanded sample of NYSE and Nasdaq stocks. The expanded portfolio is better hedged against fundamental factors and the addition of these factors has therefore less effect on estimates of the sentiment effect. Otherwise, the results are similar to those reported in Tables 13.4 and 13.5. The coefficients on B–W are consistently negative and those on Bullish consistently positive. However, there continues to be a big difference between the two periods with the coefficients being significant only in the later period.

Table 13.10 Regression of portfolio returns on lagged measures of sentiment with and without controls for fundamentals. The portfolio is an expanded long-short portfolio of NYSE and Nasdaq oil stocks (the Hi-Lo portfolio)

13.5.3 The Effect of Lagged Market Returns

To evaluate the role of market returns in generating sentiment, we augmented the VAR system by including the lagged market return in each of the regressions. This did not significantly change the relationships between either of the sentiment variables and either of the returns. It did not increase the R ²’s for the prediction of returns. The Bullish sentiment variable is not significantly related to the lagged market return in the second sub-period, where the main sentiment effect is apparent. This suggests that the momentum generated by the positive relationship between fundamental returns and lagged Bullish sentiment is not simply a proxy for the effect of lagged market returns.

13.5.4 Lagged Fundamentals

We also added to the VAR system more lags of the fundamental returns. These were generally insignificant and did not change the basic results. The sentiment variables remained significant in the second sub-period and the effect of sentiment showed up only in the fundamental regression and not the deviations from fundamentals.

13.6 Conclusions

Using a sample of upstream oil stocks where we have a good proxy for fundamental value, we show that sentiment predicts returns. However, the effect is highly time-varying, appearing only after the post-2000 increased interest in oil-related assets.

Sentiment effects come in two forms: retail investor sentiment predicts short-term momentum, and Baker–Wurgler sentiment predicts medium-term mean reversion of fundamental factors. Whilst the sentiment variables explain only a negligible proportion of the variance of returns, the additional return due to a change in sentiment is not unimportant. For example, in the second period for the portfolio of all oil stocks a one standard deviation rise in the level of investor sentiment added about .3 % to the following month’s return; during the same period a similar one-standard-deviation rise in the Baker–Wurgler index reduced return by about .3 %.

Contrary to the hard-to-arbitrage hypothesis, sentiment affects returns on these stocks through fundamentals rather than through deviations from fundamentals. Overall, it appears that retail sentiment drives the prices of oil and gas futures away from their deeper fundamental values until the deviation is sufficiently large that arbitrageurs drive the prices back towards their equilibrium values. This process for the fundamentals is then reflected in the prices of upstream oil stocks.

These effects appear even in a portfolio that is long hard-to-arbitrage stocks and short easy-to-arbitrage stocks, because this portfolio has a net exposure to fundamentals. This has implications for tests of the hard-to-arbitrage hypothesis, showing that it is important to have effective controls for fundamentals even when the long-short portfolio is used.

Our finding that sentiment affects upstream oil stocks through the fundamentals raises the issue as to whether this is also the case with other industries that invest in assets that are traded in speculative markets. Obvious examples would be stocks in other extractive industries but a similar effect could characterise financial institutions. It also prompts the question whether the magnitude of any sentiment effects depends on the extent to which the fundamentals are tradeable. If this is the case, sentiment effects might vary not just with ease of arbitrage but with the nature of the company’s fundamentals. The sharp increase in the significance of sentiment effects in the post-2000 period was accompanied by an increase in speculative activity in energy futures. If these effects are truly linked, then it raises the question as to the effect of trading activity on the influence of sentiment. These would appear to be fruitful, if difficult, areas for future research.

Appendix: Principal Data Sources

Stock samples: All common stocks with NAIC code of 211111 or SIC code of 1311 (oil production or exploration) that were listed on the NYSE or Nasdaq and whose issuers were incorporated in the USA. Returns data were taken from the CRSP monthly database. Portfolio returns were constructed from equally weighted holdings in all stocks with valid returns data for that month. Portfolio returns were then converted to continuously compounded returns.

Oil prices: Month-end spot prices for West Texas Intermediate taken from the Energy Administration website at www.eia.gov.

Natural gas prices: Monthly spot prices for Natural Gas Wellhead Price taken from Globalfindata. Prices are month averages from March 1983–December 1995 and end-of-month from January 1996–January 2011.

Contango: NYMEX futures prices are taken from Quandl at www.quandl.com. The contango is defined as the price of the contract that is sixth nearest to delivery divided by the price of the contract that is closest to delivery. The change in contango is defined as ln(contango _t ) − ln(contango _t − 1).

Baker–Wurgler Sentiment Index: SENT¹ constructed from IPO volume, IPO first-day returns, market turnover, and the market-book ratio of high-volatility stocks relative to that of low-volatility stocks. See http://people.stern.nyu.edu/jwurgler/. The index is rescaled to have mean zero and unit standard deviation.

American Association of Individual Investors (AAII) Investor Sentiment Survey: Proportion of investors reporting they are bullish divided by the total proportion reporting that they are either bullish or bearish (i.e. not neutral). Taken from final week’s survey in each month as reported on www.aaii.com/sentimentsurvey. The index is rescaled to have mean zero and unit standard deviation. Data are available from July 1987.