I'm probably going to butcher this, so here are two papers [fairly easy to read] by John Cochrane [finance guru from the University of Chicago] that should help explain finance theory and how it relates to investing/portfolios:
New Facts in Finance
Portfolio Advice for a Multi Factor World
The Capital Asset Pricing Model says assets can only earn a high rates of return if they have a high Beta
. "Beta measures the tendency of an asset to move up or down with the market. Beta measures how adding a bit of an asset to a diversified portfolio increases the volatility of the portfolio." [first paper by Cochrane]. Hence, those assets that don't move up or down with the market much [or are uncorrelated], or move opposite the market, will have low expected rates or return or do not have to demand high expected rates of return. This is why insurance usually has a negative rates of return.
Also remember that the CAPM gets its measure of risk from Markowitz and MPT. I believe that Sharpe and Markowitz helped each other out on one or both. So, portfolio volatility is the only measure of risk in the CAPM.
Fama and French [and others I'm sure] said "Whoops, there are assets whose returns cannot be explained by their beta." E.g., value stocks and very small stocks [CRSP 9-10].
What if there are other risk factors that investors care about besides portfolio volatility? For example, if value stocks do a lot worse than growth stocks in massive depressions, and investors could lose their jobs during this time, then value stocks will pay higher rates of return. Covariation of assets with recessions.
If there are other risk factors that influence people's portfolio make-ups, then the two dimensional mean-variance graph of Markowitz becomes a three or four dimensional graph [see the second paper], with expected return, portfolio volatility, and one, two, or more other risk factors on other axes.
Because if an asset class offered same or higher expected returns and lower volatility (risk of achieving that return) wouldn't we all pile in there until price got bid up?
Not necessarily. If that asset had some characteristic - say very bad returns when we were losing our job, investors may not be willing to buy that assets until it's expected return was higher [than say the market] enough [and it's price lower enough] to compensate for this risk. Investors may have been scared to death of the bad returns of value, and perhaps particularly small value, in the beginning of the century and the great depression. What if the value premium, perhaps like the equity premium, is some historical anomaly?
The value premium is still quite hotly debated in finance today.
Think about this: Consider two scenarios:
1. Value premium is compensating some risk, which perhaps hasn't fully showed itself yet.
2. Value premiums is due to people's behavioral finance issues, and it is a free lunch.
If you subscribe to #2, what if your wrong?!
So many questions.