When is a good time to invest in the stock market?

How do you know when stock market returns are going to be fantastic? Was it possible to know, in 1985, that the FTSE 100 was going to have an amazing, almost seven-fold run-up over the next 15 years? Or that in 1999 the FTSE 100 was going to go precisely nowhere for more than a decade?

I’ll start by looking at what the market is.

As I’m sure you know, the FTSE 100’s price level represents the market value of the largest 100 companies on the London Stock Exchange.

So let’s break that down into two bits. One bit is the 100 companies; the other bit is the market value.

The FTSE 100 deconstructed

The FTSE 100 is made up of 100 companies like Vodafone, HSBC and BT, some of which are vast, multi-billion pound operations which span the globe and employ tens if not hundreds of thousands of people. Most of them have existed for decades, and some of them have existed for centuries.

So keep that in mind – 100 vast companies, multi-billion pound profits, operations all over the globe, and decades of operating history.

The other side of the market is the price. This is the thing that we see whizzing up and down on computer screens or, failing that, we see it being announced on the news every evening. If it goes up or down a lot in a short period, everybody starts getting excited, and we might see pictures of traders shouting into a telephone.

The next question is – what is the relationship between these two things, these 100 companies and this FTSE 100 number we see bobbing up and down?

The answer is, not very much. There is not a particularly hard link between what the 100 companies are doing out there in the world, how much profit they’re making, how much they’re growing and how much they’re returning to shareholders as dividends and that market price.

That isn’t to say there’s no relation at all; there is. It’s why we haven’t seen the FTSE 100 fall to 1,000 in the last bear market or climb to 100,000 in the last bull market.

So the FTSE 100 is on a drunken, random walk, but with restrictions.

It’s like a drunk staggering down the road towards a pub a mile or so in the distance. The journey may zig and zag, but every time he starts to veer off into the weeds, he makes an extra effort to head back towards the middle of the road (or the pavement if he wants to avoid being run over).

The efficient market and unpredictable drunken walks

Let’s try to get back on topic with some charts from the stock market, in this case, the S&P500 from the US, because there’s much more data available for it. The data I’m using here is available from the very excellent Robert Shiller.

Here is the real (inflation-adjusted) price of the S&P over the past 100 or so years.

As you can see, the market zigs and zags around like a drunk. However, in the longer term the price is headed up and to the right because it’s loosely tied to the earnings of the underlying 500 companies, which tend to grow with inflation, plus a bit more.

Here’s another chart, this time of the real earnings of those 500 companies over the same time period.

Well, that’s pretty ziggy and zaggy too. In some years, these companies make a lot of money, but sometimes, like in the early 1930s, they hardly make any money at all. But again, you can see that, in general, the earnings march upwards, in real terms, over time. Given how volatile those earnings are, I’m not surprised that the price moves all over the place if the earnings do too.

So let’s see how well the market does at keeping in the middle of the road, in other words, at keeping in step with the current earnings of those 500 companies.

I think, by and large, I’d have to say that’s not a bad job; most of the time, the market (in red) keeps more or less in pace with earnings (in blue). There are some notable exceptions, though.

In about 1915, there was a massive spike in earnings that the market didn’t seem to notice. Then again, in the 1950s, earnings jumped up, and the market took a while to respond. Then in the 1970s and 80s, the market seemed to be lagging behind earnings, only to overshoot in the late 90’s, more or less until today (or at least 2010, which this data goes up to).

So now we know at least part of the reason why the market moves around all over the place – because it’s trying to track the current year’s earnings.

Just look at the 1920’s bubble, which ended in the 1929 crash. In terms of keeping up with earnings, the market did a good job. Earnings are shooting up, and at the same time, so is the market. Then earnings fell off a cliff into the Great Depression, and so did the market; so full points to Mr Market on that one.

This drunken walk, both of earnings and the market, is basically what investors mean when they say the market follows a random walk.

Earnings zig-zag around, and they’re not predictable. The market zigs and zags around somewhere near some multiple of the earnings (in the chart above, the market plot is set at 12.5 times the earnings), and that’s also not predictable.

That’s why academics mostly say that we should just buy the index and settle for the market rate of return, but I want more than the market rate of return.

Predicting the drunken walk using average earnings

Perhaps there is some way of cancelling out some of the noise so that we can get a clearer picture of what’s going on?

One way to do this is to look at earnings over a longer period than just the last year. There is no magic amount, but a popular smoothing technique is to average out the real earnings over the previous ten years and compare the earnings to that.

So what does this real 10-year average earnings look like?

Wow, that’s about the most boring chart ever. However, it does show that even over 10-year periods, the earnings of a national index can go down, in real terms, for a long period of time. Just look at the 1920 to 1940 period. But it’s a whole lot smoother than just the yearly real earnings, so perhaps this might give our drunken market something a little easier to follow.

Surely if the market was focused on this 10-year earnings period, it wouldn’t zig and zag around nearly as much as it usually does? Let’s put the two together.

I’m no rocket scientist, but that definitely looks better to me. The green line (real 10-year average earnings) provides a much clearer signal, or middle of the road, for the market to follow. Once again, the market seems to do a pretty reasonable job of tracking earnings over a century. But, again, there are some notable exceptions.

This time the first exception is around 1920. The market falls off a cliff and is miles away from its normal multiple to the 10-year average earnings (the Cyclically Adjusted PE, or CAPE ratio for short). This is a clear signal that the drunk (the market) is well off track and should revert back to the middle of the road. And that’s exactly what it does.

Between 1920 and 1926-ish, the market had an incredible run-up, with huge returns from that low point which was clearly visible to anybody who had been looking at the CAPE ratio at the time.

The next exception is the 1929 peak. This time the market is way above the earnings line, and again it’s a clear signal that something needs to change… and it does. This time the market crashed, and all those who were in the market at those high valuations got burned very badly.

The interesting thing here is that by looking at the market price relative to one-year earnings, there was absolutely no warning signal at all. It’s only by looking at the CAPE ratio that it becomes clear something was seriously wrong.

This pattern happened again in 1932 when the market was way off course. Then it was clear that the market may have been overvalued in the late 60s, undervalued in the late 70s and early 80s, and very overvalued in the late 90s and early 2000s.

Some of this is visible with the one-year data, but not to the same extent and not as clearly.

And so here’s the point:

When the market is ‘undervalued’ relative to “smoothed earnings”, i.e. when the CAPE ratio is low, the market is more likely to go up and therefore future returns are likely to be higher than normal.

The standard way to measure asset classes is by returns and the variability of those returns. However, very few people seem to take note of how those returns vary over time and how it might be possible to spot, beforehand, when future returns are more likely to be higher than normal.

With hindsight, it’s clear that S&P 500 returns from 1932 were spectacular, while returns from 1929 were terrible. In real terms, it took some 30 years to break even from 1929, while the returns in the same period from 1932 were about 500%. That’s a pretty big difference.

By looking at the CAPE ratio at the time, you could have come to the same conclusion, which means you may have been able to sidestep the 80%+ losses in the market by getting out of equities when they were expensive.

Even better would have been to get back into the market in 1932, when it was obviously cheap. But it was only really obvious if you were looking with the right tools, and the CAPE ratio is a much better tool than the standard PE ratio.

So when is a good time to invest in the stock market?

The FTSE 100’s CAPE has averaged around 15 over the long term. At the peak of the dot-com bubble, it stood at 30, and future returns from that point were terrible. In the depths of the 2009 bear market CAPE stood close to 10, and returns from that point have been spectacular.

As a general rule of thumb then, I would say:

• When CAPE is below 10, returns are likely to be exceptional
• When CAPE is close to 15, returns are likely to be “normal”
• When CAPE is higher than 20, returns are likely to be weak and
• When CAPE is higher than 30, returns are likely to be terrible