That said, why would someone believe the future is more unlikely to be like the past than it is to be like the past?
I think the issue here is what does it means to be "like the past"? There are multiple ways of defining "like" that could lead to widely different forecasts.
For example, consider the following two approaches
(1) You believe that the future stock returns will be like the equally weighted average of returns in the past. So to create your expectation you take a simple average and use that as your forecast.
(2) You believe that future stock returns will be like past years when the market had similar Schiller P/E values (or whatever market condition you think is important). So to create your expectation you take a weighted average (giving higher weight to past years with similar Schiller PE) and use that.
In both cases, we are trying set expectations based on the idea that the future will be like the past. It's just that the definition of "like the past" is a little different in each case.
Some day, the people who say "this time it's different" may well be right. But what empirical evidence to we have that *this* time we say it will be different than the many other times we've said it (as recently as about 2009)?
In some cases like with Schiller PE there's a mountain of evidence that this is a real effect and not some spurious result. Other methods (like using the butter production in bangladesh to predict S&P 500) may have more limited support.
So while the past isn't gospel and isn't proof of what the future will be, I think it's a better (if imperfect) guide to the future than some scenario that does not resemble anything in history. By all means, hedge your bets, but....
The people who make models to forecast stock returns are all basing it, directly or indirectly, on past data. The model might produce different predictions than the historical average but that's because current conditions are not historically average.
That said, I have an AA set to 55-60% equities, and I plan for about a 2.5% real rate of return in the long run, but also run the numbers with a 1-1.5% real rate of return as a sanity check. If we get less than a 1% real rate of return over the next 30-40 years, we're all a lot more screwed than just our portfolios.
I will not be happy either with such low rates of return. But forecasting is extremely difficult and even the best models (which tend to be pessimistic now) do not exclude good outcomes when you look at the range of expected results instead of a point estimate.