Market behavior is near-random*. That's settled science.

[OldShooter]

@Lsbcal's request, I am starting this thread with a post I made in another thread plus a little amplification. It was a response to a post of his where he seemed to assert that a belief in the random nature of the markets was a sort of religion.

---Quote (Originally by OldShooter)---
Hmm ... that may be key to the discussion. I think the near-random behavior of the markets is settled science, not even close to being religion.

It is fundamental to Modern Portfolio Theory, which has been around for sixty+ years and got Markowitz his Nobel prize. (It makes no sense to calculate variance for a variable that is not random.)

S&P's semiannual Manager Persistence Report Card results, which consistently show a lack of persistence, are completely consistent with a random market.

There is also ample witness testimony: https://www.bogleheads.org/wiki/Taylor_Larimore's_market_timing_quotes

Is there serious academic research or statistical evidence to disprove any of this? (Anecdotes don't count.) I would love to see it, because it might give me a way to beat Mr. Market.
---End Quote---

Additions:

Nobel prize winner Eugene Fama and his research partner Kenneth French studied thousands of asset managers and were unable to find any evidence that their results were anything other than random. Short video here: https://famafrench.dimensional.com/videos/identifying-superior-managers.aspx Their published paper on this ran to 46 pages of mathematics IIRC -- more than I could handle but YMMV.

Common Objection: "I know a guy." or "What about this guy?" ... who successfully forecast the 2008 crash and several subsequent market gyrations.

The market is "noisy" and there are thousands of people making forecasts. When something happens in the market, it is inevitable that some of these forecasts will have been correct. It is also inevitable that some forecasters will be correct several times. There are about 10,000 mutual funds in the US; if we set 10,000 monkeys to flipping coins, after the 8th toss about 40 of them will have flipped ten heads in a row. Others will have had near-perfect records, but the overall average will be what the random nature of coin-flipping predicts: about 50%. Focusing on the luckiest monkeys is misleading when attempting to understand the game.

*I say "near-random" because the market were completely random (classical Gaussian distribution) the deviations would be centered on zero and there would be no point in investing because the markets would never change. Actually the distribution is centered a few percent to the right/positive, resulting in what we have seen in a hundred years of market history: a slow, steady trend interrupted frequently by random excursions. That's why buying and holding a diversified portfolio works. The buy and hold investor basically believes in and rides the trend, ignoring the noisy excursions.

Finally, I don't claim that any of this is original with me. I was trained as an engineer and scientist, so I go looking for data any time I have a question or want to understand something. I am reporting here on what I found. What I did not find was any data that refutes the randomness assertion.

 

[HI Bill]

Just opinions and thoughts here, not backed up by study or statistics. Geologists are always looking for patterns, and associations, and cause and effect. I don't believe the markets are truly random, but I do believe it's more like a freeway with many cars, and many unrelated events. One car slows down, for whatever reason, and traffic backs up. Then it takes a while to get back up to speed whenever the car moves on.

The markets are driven by fear, greed, political events, and natural events. It's too complex to find patterns with solid cause and effect relationships, and impossible to predict. So, I guess random is as good of a description as any. Occasionally, things live a government incurring too much debt actually have a direct impact on the markets.

When the current president was president-elect, the markets started rising, even before he took office. It's the fear and greed of people who drive the markets, more than anything. And that, is almost certainly not predictable.

 

[Spock]

I think Greenspan copped out when he referred to "animal spirits" driving the markets.

For the last 10ish years, the central banks have been actively propping up markets by suppressing interest rates which drove massive buybacks and sent those who normally stick to less risky assets to "reach for yield" and take on risks they normally wouldn't. While there is noise on the 10 year curves, the overall curve nothing close to random.

 

[OldShooter]

Just opinions and thoughts here, not backed up by study or statistics. Geologists are always looking for patterns, and associations, and cause and effect. I don't believe the markets are truly random, but I do believe it's more like a freeway with many cars, and many unrelated events. One car slows down, for whatever reason, and traffic backs up. Then it takes a while to get back up to speed whenever the car moves on. ...

I think you're pretty much on target. Consider a trout stream. No one would expect to find an expert who who could predict all the vortices and ripples. They appear to be completely random yet we actually have the mathematics. But the problem is too complicated so we are stuck with "random." Same-o with markets. Maybe if we modeled every microeconomic transaction and measures all the behavioral influences we could say they are deterministic. But that's impossible so again we are stuck. (I think chaos theory applies here but I am not familiar enough to really understand it.)

I think Greenspan copped out when he referred to "animal spirits" driving the markets. ...

Richard Thaler believes Greenspan was prescient, talking about what is now called behavioral economics and effectively rejecting the purest form of the Efficient Market Hypothesis. Thaler also refers to a chapter in one of Keynes' books and says that he might be considered to be the father of behavioral economics.

For the last 10ish years, the central banks have been actively propping up markets by suppressing interest rates which drove massive buybacks and sent those who normally stick to less risky assets to "reach for yield" and take on risks they normally wouldn't. While there is noise on the 10 year curves, the overall curve nothing close to random.

Oh, lots of things are clear in the rear view mirror. Thaler (again) discusses the fact that humans want reasons for things, so even make them up as necessary. Nassim Taleb spends quite a bit of time on this, too.

For example, we can easily see that the 737Max had a major impact on Boeing stock, but that does not mean that anyone could have forecast the path the stock has taken in the last couple of years.

 

[marko]

I may be out of my depth here but to me over time the market is driven by profits, no? Profitable companies drive the market higher and less profitable ones drive it lower.

Fundamentals.

There are temporary panics and temporary exuberances of course, but over the long-haul, it is a company's profitability that makes someone decide if it a good stock/fund to own.

If I understand OPs point of randomness, one could do just was well with a dartboard but AFAIK, it doesn't reliably work like that. A buggy whip company would have an equal chance of winning as would MSFT, but it doesn't.

Again, I may be misunderstanding the point here (wouldn't be the first time today).

 

[HI Bill]

I may be out of my depth here but to me over time the market is driven by profits, no? Profitable companies drive the market higher and less profitable ones drive it lower.

Fundamentals.

With mutual funds, ETFs, and 401(k)s, I think there's a lot more in play here than profits. Look at Bitcoin...produces nothing, backed by nothing...varies according to ____? Much of the equities market is a casino built on fear, faith, and greed.

 

[ExFlyBoy5]

I just don't take any credence with any "forecasts" of what the market it going to do. There are simply too many players and variables to come up with any real idea of future performance. Yes, we often look at the past for future possibilities, but the future is NOTHING like what the past was. For the most part, the average person has ZERO idea what our society will look like/act like in the next 10-20 years and there just is no good way to predict (with any reliability) what the markets will do.

My simple method is: Hope for the best and plan for the worst.

 

[marko]

With mutual funds, ETFs, and 401(k)s, I think there's a lot more in play here than profits. Look at Bitcoin...produces nothing, backed by nothing...varies according to ____? Much of the equities market is a casino built on fear, faith, and greed.

True. One can always find several exceptions and there are always the ones that confound logic. I remember when everyone was trying to figure out how Google was going to make money being a free search engine. Then there was WeWork.

Market behavior is different from what a single stock does IMO.

But again, over time, I just see most investors (especially the big boys) making more rational decisions based upon some sort of fundamentals, be they profit or potential for profit.

If you're a day trader, sure. There's a ton of randomness, faith and fear there. If you're a long term investor as you might be with a MF, I think (hope) there's more thought going into it.

 

[Alexi]

Don't put all your eggs in one basket

 

[NW-Bound]

The market movement is of course random. What's new?

Physical events around us are all random. Even a determinitic system can be practically viewed as random, when an infinitesimal disturbance leads to a large deviation in the future. See the seeemingly simple Lorenz equations. We would need computers with infinite precision, not the current 64-bit CPUs we now have, to compute these equations.

And even in math, which is called the Queen of the Sciences, uncertainty lurks there too. See Gödel's Incompleteness Theorems.

However, many randomnesses are bounded, in nature as well as in human endeavors. We cannot predict when the next hurricane will hit Florida, but there is no risk of it wiping out the entire US (however an asteroid could). We cannot know when the market will return 30% in one year, but we know that if one already occurs this year, then the chance of it happening again next year is not that good. That does not mean that it cannot happen. A lot of crazy things has happened. It's all about likelihood.

It is not the acknowledgement that randomness is the nature of things that seems religious. It's the absolute attitude that one can do nothing, and should do nothing, not even think about what might be possible.

I cannot predict when the next hurricane will hit Miami, but if I see a storm barreling down towards me from 500 miles away, I will get the heck out of there. It may still veer off at the last minute, but we are talking about likelihood here. Similarly, exact prediction of the market is impossible, but trying to guess the range of outcome in the near future is not so futile, in my opinion. It's still damn hard though. In fact, prediction of the economy often turns out to be more right than prediction of the market. The two can be decoupled.

 

[LOL!]

There is no way the market is random. Sorry, but if I was a very important person such as the dictator of a small Asian country and bought some of the appropriate options, then sent a message to US news outlets that I had just parked a small submarine with a small nuclear device in the NY harbor and was asking for one million dollars, then I think the stock markets would react predictably in my favor at least for a little while.

Of course, random investors would not know this was going to happen, so they would not benefit ahead of time.

 

[NW-Bound]

^^^^ Hmmm... You've got a point.

I forgot about the Dr. Evil scenario, which is not laughing matter as it is in the movie scene below.

 

[UnrealizedPotential]

There is no way the market is random. Sorry, but if I was a very important person such as the dictator of a small Asian country and bought some of the appropriate options, then sent a message to US news outlets that I had just parked a small submarine with a small nuclear device in the NY harbor and was asking for one million dollars, then I think the stock markets would react predictably in my favor at least for a little while.

Of course, random investors would not know this was going to happen, so they would not benefit ahead of time.

Maybe, but how would you know how long this will play out? What if this is resolved fairly quickly? How would anyone account for events beyond their control?

 

[bmcgonig]

It is not the acknowledgement that randomness is the nature of things that seems religious. It's the absolute attitude that one can do nothing, and should do nothing, not even think about what might be possible.

.

Exactly.

French himself admits that there are managers that can best the market, just that they're too hard to find, and if you do find them everyone else will too so it will even out.

 

[NW-Bound]

Maybe, but how would you know how long this will play out? What if this is resolved fairly quickly? How would anyone account for events beyond their control?

While I cannot tell when such a dictator can sneak a bomb into NY harbor, if and when such event happens and the market crashes, I will step in to buy.

If the world goes down the proverbial chute, what's the point of keeping cash instead of stocks? I've got little left to lose. But in the chance of it recovering, I will make out well.

Hence, while predicting events is not possible, once they happen, our reaction to them still makes a difference.

I am always thinking more about reacting, rather than predicting.

 

[Lsbcal]

@Lsbcal's request, I am starting this thread with a post I made in another thread plus a little amplification. It was a response to a post of his where he seemed to assert that a belief in the random nature of the markets was a sort of religion.
...

Hi OldShooter, I did not assert that a "belief in the random nature of markets was a sort of religion". I was referring to the continuous comments of some posters that seem to feel the need to bring us back to their way of thinking, i.e. any form of market timing is not acceptable. If you like you can strike my "religious zealotry" sentence from the record ... objection sustained your honor.

Anti-market timing investment philosophy is quite alright in my opinion but not when it interrupts thread flow and becomes a pro vs con debate on market timing IMO. I even said in that same thread that buy-hold is probably the best investment approach for most people. I was hoping that would help to diffuse the OT direction the thread was heading.

Now I do think the markets are pretty random in that future data is clearly unknown ahead of time and continuously drives prices either up or down. We cannot know what the new data (or events) will show and all we have are guesses before the actual data arrives.

I do think that there is one area where markets are not 100% random. That would be in the general area of "momentum". Probably this exists because not all market participants immediately react to new data. Some react to raw data immediately using perhaps computer driven algorithms, some react somewhat later perhaps in hours, and some react to the reactions which can have echos I think going on for days or even weeks.

 

[OldShooter]

I may be out of my depth here but to me over time the market is driven by profits, no? Profitable companies drive the market higher and less profitable ones drive it lower. ...

Pretty much yes. The way I think about the long term market trend is to say that I have hired a very diverse group of employees that, on balance, are working away to generate cash flow that is my payback.

If I understand OPs point of randomness, one could do just was well with a dartboard but AFAIK, it doesn't reliably work like that. A buggy whip company would have an equal chance of winning as would MSFT, but it doesn't. ...

Almost. Randomness doesn't say who will win or lose. Rather it says that all dartboard portfolios with size enough to be diversified will produce the same long-term results.

... But again, over time, I just see most investors (especially the big boys) making more rational decisions based upon some sort of fundamentals, be they profit or potential for profit.

If you're a day trader, sure. There's a ton of randomness, faith and fear there. If you're a long term investor as you might be with a MF, I think (hope) there's more thought going into it.

There is no evidence that the professionals do any better picking stocks than the amateurs. Actually it might be the other way around because the professionals are burdened by costs that the amateurs do not have. In addition to the S&P Manager Persistence report S&P also produces a semiannual "SPIVA" report comparing professionals' results to their benchmarks. Over a year maybe a quarter to a third beat their benchmarks; randomness would predict this. Over five or ten years typically under 10% have been lucky monkeys long enough to beat their benchmarks. Morningstar's take on the situation is pretty much the same as S&P's: https://www.morningstar.com/article...might-help-investors-improve-their-base-rates

... I cannot predict when the next hurricane will hit Miami, but if I see a storm barreling down towards me from 500 miles away, I will get the heck out of there. It may still veer off at the last minute, but we are talking about likelihood here. Similarly, exact prediction of the market is impossible, but trying to guess the range of outcome in the near future is not so futile, in my opinion. It's still damn hard though. In fact, prediction of the economy often turns out to be more right than prediction of the market. The two can be decoupled.

Well, hurricanes are a little different because they can be identified early and forecast. The 737MAX, the 1987 market crash, and Enron more typify things that affect the market. Re economic forecasting I think you would enjoy Nate Silver's chapter on that subject in his book "the signal and the noise." His conclusion is " ... economic forecasts are blunt instruments at best, rarely being able to anticipate economic turning points more than a few months in advance. Fairly often, in fact, these forecasts faile to "predict" recessions even once they were under way. ...

There is no way the market is random. Sorry, but if I was a very important person such as the dictator of a small Asian country and bought some of the appropriate options, then sent a message to US news outlets that I had just parked a small submarine with a small nuclear device in the NY harbor and was asking for one million dollars, then I think the stock markets would react predictably in my favor at least for a little while.

Of course, random investors would not know this was going to happen, so they would not benefit ahead of time.

I don't think anyone could credibly claim that the market is not affected by events and sentiments. The reason the resulting path is random is that the events and their timing is unknown until they occur.

Your theoretical dictator would be trading on inside information and, for him, the market would be somewhat predictable. But for other players it would still be random.

Jeez, as I have been typing this a lot of good posts have appeared. I'm going to sit back and read for a while.

@Lsbcal, if I misunderstood you I'm sorry. But good idea anyway to suggest this thread. Re market timing I have been puzzled by the sense here that it is "not acceptable." Anything the investor wants to do is "acceptable." It is his/her money. I will only go so far as to say that if the market is essentially random, trying to time it is probably futile. But I don't understand giving it a moral dimension.

 

[NW-Bound]

... I do think that academic research has pointed out one area where markets are not 100% random. That would be in the general area of "momentum". Probably this exists because not all market participants immediately react to new data. Some react to raw data immediately using perhaps computer driven algorithms, some react somewhat later perhaps in hours, and some react to the reactions which can have echos I think going on for days or even weeks.

Immediately after the election day in 2016, I saw that ITA, an aerospace & defense ETF, jumped more than the broad market. It was presumed to be due to the anticipation that the new administration would be more supporting of this industry.

I did not buy, because I thought it was already old news and no more gain could be had, according to the "market efficiency theory". Nope. ITA kept on leading the market for 2 weeks, before the difference between it and the S&P was erased. So, being one day late I could still make some money.

Not all investors react immediately as you think. There are a lot of johnny-come-lately. You do not have to be the first. Just do not be the last.

 

[NW-Bound]

... For example, we can easily see that the 737Max had a major impact on Boeing stock, but that does not mean that anyone could have forecast the path the stock has taken in the last couple of years.

I have not held Boeing stock for 2 decades, but if I held it recently, would have lightened up when the 2nd crash occurred.

Yes, I would not know what came after that crash, but given the run-up in Boeing stock in the recent years, a lot of anticipation of higher profits was already baked into the stock price. If things worked out, the stock may still go up a bit more. But in the chance of a more serious problem, there would be hell to pay.

Again, not really predicting, but looking at the possibilities of how things may pan out, and going for a safer course of action.

 

[target2019]

There's no "settled science" as the thread title proclaims.

See https://en.wikipedia.org/wiki/Random_walk_hypothesis

Economic theory, may be proved in the future.

 

[OldShooter]

There's no "settled science" as the thread title proclaims.

See https://en.wikipedia.org/wiki/Random_walk_hypothesis

Economic theory, may be proved in the future.

The title is careful to say "near-random." Proof that the market is not perfectly random is as simple as observing that the distribution is biased, which I did in the post. The existence of momentum, also pretty well settled science, also proves that it is not perfectly random. The Andrew Lo book, which I have, also serves.

The point is that not-quite-perfect randomness is proven well enough to explain most of what we as investors see and the decisions we have to make. Inability to time the market is probably the most important. It's the other side of the predicting-the-market coin.

I have deliberately avoided the Efficient Market Hypotheses because it is not necessary to support the assertion and it is hardly a settled matter. (For those interested, here is a 40 minute video that I enjoy and learn from every time I look at it. Two Nobel Prize winners discussing the topic. https://review.chicagobooth.edu/economics/2016/video/are-markets-efficient)

That is not to say that it is impossible for little investors to nibble around the edges of imperfect randomness and maybe get a chunk of cookie from time to time. @NW-Bound has mentioned a few. Occasionally winning a momentum bet, taking advantage of the market's imperfect reaction to earning announcements, and occasionally succeeding in exploiting an overreaction to an event. This is sort of protected territory for the little guy IMO because his tiny money does not move the market. A professional running a significant portfolio must make large enough trades that they will reduce or eliminate his success even if he sees the same little things that the little guy sees. Which he probably does.

But the tricks that a small investor might successfully execute don't, I think, matter for the vast majority of investors on this site because they are not equipped or inclined to do them. For that majority, accepting real world randomness should help them understand why betting on sectors increases risk, why chasing successful managers is doomed, and why the best policy is to completely ignore the talking heads, click-seekers, and hawkers of expensive mutual funds.

 

[DrRoy]

Over time the market is driven by profits, no? Profitable companies drive the market higher and less profitable ones drive it lower.

Fundamentals.

I agree in general, but would revise it to say the perceived value of profits. This year is a good example. S&P 500 profits are nearly flat, but the index is up sharply. The perceived value of the flat profits is higher due to lower interest rates, and the belief that they will rise in the future as the trade war is resolved and recession is avoided.

 

[donheff]

I'm in the near random with a biased (+) direction impacted by the butterfly effect camp. Keep a diversified portfolio and generally don't try to time the market (e.g. go heavily toward fixed because we are at market highs now). But if an obvious monster butterfly hits (like a big earthquake trashing the west coast) and the market tanks in panic I would probably try to jump way toward the equities side. Still no guarantee and no specific way to gauge the degree and timing of the bounce back but...

 

[GenXguy]

There is no way the market is random.

Agreed. It's not random!

 

[OldShooter]

I'm in the near random with a biased (+) direction impacted by the butterfly effect camp. ...

Yes. As you implicitly point out, my trout stream metaphor in the OP was incomplete. We do understand the mathematics of the fluid dynamics but the more complete metaphor would include people on the shore, like @LOL!'s dictator, throwing rocks into the stream.

Agreed. It's not random!

OK, your evidence, please? Statistical studies, academic research, etc. similar to what I had in the OP, not anecdotes.