Big open puzzles that haven't been resolved yet.
The puzzle of excess volatility is the apparent paradox that broad market stock indices such as the S&P 500 are much more volatile than would be justified by the volatility of the underlying discounted dividend flows. It extends to other asset classes as well, such as foreign exchange and corporate bonds, but for now I’ll focus only on the stock indices.
Naively, we might expect that given a stock which pays a certain cash flow (including both dividends and buybacks) to the investors, the price of the stock would be the present value of these cash flows discounted at some rate of return. If this were the case, volatility in stock indices would only be driven by volatility of expectations about future dividend flows. However, under this picture it’s difficult to justify the observed volatility of stock indices, which is 15% annualized in good times and can go up to 80% annualized as we have seen in March 2020. This is because while there can be substantial uncertainty about current dividends, what dividends will be this quarter or this year should not have much effect on the stock price under a discount rate formalism.
A little financial-econometric history by John H. Cochrane summarizes the history of the puzzle and confirms the above reasoning. I highly suggest you read it, since it contains a lot of equations that I am unable to display here due to GitHub Markdown’s lack of support for LaTeX. The upshot is that the data suggests all of the variation in dividend yields of broad market stock indices corresponds only to variation in future returns, with none of it corresponding to variation in future dividends. In other words, if prices are high today relative to dividends today, that doesn’t mean dividends will grow faster in the future or that prices will grow faster in the future - it only means that you’ll get, on average, a worse return that you would’ve otherwise got. The reason the naive argument fails is that it assumes rates of return don’t vary over time, while variation in returns is the primary driver of the phenomenon of excess volatility.
Of course, from one perspective this does nothing to resolve the puzzle because we can always, after appropriate log-linearization of the definition of returns, express the current price-dividend ratio in terms of future dividend growth, future returns and future price-dividend ratios. A true resolution of the puzzle would also include an explanation of why returns on the stock market vary by such huge amounts, and it’s the explanation for this fact which has proved elusive.
What explanations have been offered for this puzzle? There are many, though none of them are satisfactory. John H. Cochrane’s literature review Macro-Finance contains the following zoo of explanations, along with some exploration of the problems with each of them:
These also don’t exhaust all explanations that have been proposed: there is a whole separate class of explanations, often advocated more by high-frequency traders and people who work in quantitative finance, based on trading mechanics. For example, The Inelastic Market Hypothesis by Jean-Philippe Bouchaud proposes a diffusion model of the order book with limited memory, in which orders can have a permanent impact on the market price of an asset as expectations realign around the new market price.
The appeal to strictly mechanical explanations is not entirely unreasonable given how inadequate economic explanations have been so far. The basic problem is that to get the observed large volatility of stock indices and the sizeable equity premium, we need to have expected returns which are both high on average and also vary a lot across time, which in economic terms turns out to require a large variance and a large heteroskedasticity of the marginal utility of consumption (also called the stochastic discount factor process in the literature). The difficulty is finding a model which explains the puzzle of excess volatility without making out of line predictions of other asset price data.
I think Cochrane does a good job summarizing the problems with the economics approaches I listed above, so I won’t write my own criticisms in this page - I’d mostly be repeating what he already says in his review. If you’re curious, you should read his literature review.
What about the approaches based on trading mechanics? I think the important test here is the following: all of the macroeconomic models predict that excess volatility is caused by heteroskedasticity of the marginal utility of consumption, and that it will only be observed in assets which have a correlation with the consumption process - assets such as stocks, bonds and foreign exchange, to name a few. The economic models are not vacuous: they predict that in the absence of such a correlation, there should be no excess volatility observed in a market, at least at low enough frequencies so that one can abstract away from the specific mechanics of a particular market. In particular, if excess volatility were to be observed in sports betting markets, it would be a direct refutation of the macroeconomic approach to explaining the puzzle of excess volatility. To my knowledge, however, all excess volatility in sports betting markets is exhibited at high frequencies and is not observable at lower frequencies.
Of course one can object by saying that betting markets are way too small for a particular mechanical explanation to be operative, but I think it is a strength of the macroeconomic approach that there is a pattern of excess volatility that it would be unable to explain, even in principle, and that no such excess volatility is observed in the data. I think this observation also poses a problem for behavioral or probability distortion based explanations, since there’s no particular reason to expect people to be much more rational and level-headed when betting on sports events featuring their favorite teams or players compared to when they are buying SPY shares.