Morningstar’s (MORN: $82/share) ETF and mutual fund ratings have been the go-to standard in the fund industry for decades now, and the company does an impressive job of providing research on a massive number of funds.

Millions of investors rely on Morningstar’s research. Some would say there is so much reliance that changes in Morningstar ratings have a significant impact on mutual fund money flows. Many have observed accelerated growth in assets under management (AUM) for funds that get upgraded to a four or five star rating from MORN. Few would attempt to argue that Morningstar’s influence is formidable. The company sets the standard for marketing research services.

And it appears this marketing really pays off as a Vanguard study in 2010 showed little evidence of outperformance for five and four star-rated funds. In other words, Morningstar’s ratings drive investors’ investment decisions even though the ratings do not appear to be very indicative of future performance.

Perhaps, this disconnect is due to the fact that MORN’s fund ratings are almost entirely based on the past performance of ETFs and mutual funds. The investment quality of the holdings of funds hardly comes into play. I find this ironic because the performance of a fund is only as good as the performance of its holdings, and a rating that doesn’t take into account the holdings is not really predictive.

In addition to looking at a fund’s track record, one needs to look at its current holdings to gain insight into future performance. That’s why our predictive fund ratings are based on our stock ratings of the fund’s holdings as well as the fund’s total annual costs.

Figure 1 shows several examples from recent years where New Constructs and Morningstar have had contradictory ratings on funds. In most of these cases, Morningstar has had a positive rating and New Constructs has had a negative rating, as Morningstar’s ratings are overwhelmingly biased towards positive ratings. However, there are also cases where Morningstar’s ratings were more negative than ours.

Figure 1: Comparison of Ratings Performance

Morningstar

Sources: New Constructs, LLC and company filings

These are just a few examples, as our access to Morningstar’s ratings history is limited. They don’t show a comprehensive pattern of over or underperformance, but they do give specific examples of why investors should consider incorporating research other than Morningstar’s ratings in the process.

In the cases where Morningstar explains the methodology behind its ratings, fundamental analysis is either absent or somewhat lacking in diligence. In Morningstar’s article recommending Vanguard Small Cap Value ETF (VBR), the fundamental view section contains 1) A track record of small-cap value stocks overall, 2) a paragraph about the U.S. economic recovery, and 3) a price to earnings ratio.

When that article was published on Morningstar, we had a 2 Star, Dangerous rating on VBR based on our ratings of the ETF’s holdings. Taken as a whole, its holdings had a price to economic book value ratio above 3 and a market implied growth appreciation period (GAP) of roughly 40 years. These metrics showed that the fund’s holdings were not as cheap as a simple P/E ratio would suggest.  Sure enough, VBR has lagged the Russell 2000 index since April 2013.

Past performance clearly has a strong effect on Morningstar’s ratings, even though countless research has shown that past performance is not a good predictor of future returns. For instance, when American Beacon Small Cap Value Fund (AVFIX) earned Morningstar’s 4 Star rating in January of last year, it had a prior 3-year return of 98%.

Once again, our analysis of AVFIX showed a much less rosy picture, and once again it has significantly lagged its benchmark over the past year.

Even when Morningstar’s ratings do look past prior performance, such as their recommendation of LongLeaf Partners Fund (LLPFX) in February 2012, the fundamental analysis is lacking. Even though LLPFX had been basically flat the year before, Morningstar recommended the fund based on the quality of the managers and their “skin in the game”, the fact that they invested their own money in the fund.

However, research into the holdings showed that the average return on invested capital (ROIC) of companies held by LLPFX fell right in the middle quintile of all companies, and they were overvalued based on GAP and free cash flow yield. In the two years since, LLPFX has returned less than half as much as the S&P 500.

Looking Forward

One of Morningstar’s recent ratings that we disagree with is its 5 Star rating of First Trust Consumer Staples AlphaDex (FXG). The Consumer Staples sector is our top-rated sector, but FXG is not one of the best ETFs in the sector and only earns our Neutral rating. Other ETFs such as SPDR Consumer Staples Select Sector ETF (XLP) and Vanguard Consumer Staples ETF (VDC) look much more attractive to us in this sector. More on the best and worst Consumer Staples sector funds is here.

Over 60% of FXG’s assets are allocated to Neutral-or-worse rated stocks, and it has above average total annual costs of 0.78%.

Constellation Brands (STZ) is a top five holding in FXG at 5.4% and earns a Dangerous rating. The wine distributor has seen some recent growth due to its expansion into the beer industry, but its return on invested capital (ROIC) still stands at an unimpressive 7%. Investors are probably aware of STZ’s $3.8 billion in adjusted total debt, but its $900 million in employee stock option liability not as well publicized. The truly dangerous aspect of STZ, however, is its valuation. The current valuation of ~$84/share implies that the company will grow after-tax profit (NOPAT) by 21% compounded annually for the next nine years. The wine and beer industry is not exactly a high-growth market, so these expectations seem unreasonably high.

Allocation to Dangerous stocks such as STZ make FXG only a Neutral ETF, and certainly not one worthy of a five star rating. As you can see from Figure 1, our Neutral rating on Allianz Mid-Cap Value Fund (PQNAX) correctly predicted that it would match its index’s performance, even though Morningstar had assigned it a 2 Star rating. In this case, we expect FXG to perform similarly to the market and lag behind many of its fellow Consumer Staples ETFs. FXG has put up impressive returns in recent years, but its holdings are overvalued compared to its peers.

Morningstar’s ratings hold significant value to investors as they give insight into the past performance and strategies of fund managers. Understanding what a fund has done in the past is important, but it doesn’t tell the whole story. Even managers who have outperformed for over a decade can suddenly go cold. To really understand what the future holds for a fund, in-depth analysis of holdings is key.

Sam McBride contributed to this report.

Disclosure: David Trainer and Sam McBride receive no compensation to write about any specific stock, sector, or theme.

Feature Photo Credit: Rich Murray  (Flickr)