The father of value investing, Benjamin Graham, explained this concept by saying that in the short run, the market is like a voting machine--tallying up which firms are popular and unpopular. But in the long run, the market is like a weighing machine--assessing the substance of a company.
So in the long run yes the market is efficient, but for years on end it isn't. We are a society that likes fads and stocks are no different than hemlines, or boy bands. Some fashion buyers are better at anticipating which way fashions will go than others, and some record producers are great a figuring out which boy band will appeal to teenage girls. I suppose it is possible that this all just random chance, but when I hear these people interviewed they always strike me as being smart, and very insightful.
I think one of the big disservices, that Bogleheads community has done is over generalize from the oft quoted statistic that only 25% of actively managed funds beat their index. The performance of active funds increased
to 40% over the last five year period.. While it certainly makes sense that on average active management is going to under perform indexes. It doesn't follow that just because most funds can't beat indexes, individuals can't.
Mutual funds are not individuals. They generally are run by teams of people, and supported by others (e.g. researchers), the team member change often in the same way that the coach staff for a pro team changes a lot. A good offensive/defensive coordinator can make a good coach look bad and vice versa. None of the research on mutual fund performance I've seen tracks even head managers much less everybody on the team.
Mutual funds suffer from significant disadvantages that individual don't.
Several comments on this thread talk about the unique advantages Warren Buffett has. While these are true, Buffett suffers from a huge disadvantage that rest of us don't, he has to buy so much stock to make a difference that his buying and selling moves the market. For instance it would take 35 days for Buffett to sell his stake in Wells Fargo at WFC normal trading volume if he was the ONLY seller. Even a typical actively managed mutual funds like Wellesley has a day supply of stock for its top holdings. If you look at Warren his greatest out-performance occurred early in his career long before he was household name.
Second mutual fund manager decisions to buy and sell are dictated at the whim of the masses. I suspect very few active money manager were anxious to sell stock in early 2009, yet they had no choice due the masses of people redeeming their shares. I am pretty sure that hedge funds and probably closed end funds did better simply cause they didn't have to
redeem shares at the market bottom. But again these money managers aren't tracked by academics.
Some stock pickers are good at figuring out which stocks are going to be popular in the short term, and others are good at assessing the true long term value of a company. And yes a significant chunk of these people are just lucky, same thing is true for record producers and buyers. Still when I listen to the average person talk about investing or even plenty of stock brokers, financial advisers and such. I say to myself I know more than these people. Then when I read many of the folks on the board I say these folks are smarter than I am about this stuff. I started tracking my investment pretty careful shortly after I retired and in general about 1.3% above 75/25 mix, lately Schwab has been telling me I'm doing it with less risk. I wouldn't be at all surprised if many on the board are doing even better.