Tuesday, June 7, 2011

Rating Consumer Reports New Car Ratings

Consumer Reports is one of the most trusted names in new product reviews and testing.  They pride themselves on being fair, thorough, and independent.  As a non-profit organization that accepts no money, advertising, or freebies from companies, they can tell it like it is.  Millions of people rely upon their unbiased advice when it comes to making purchases, both major and minor.  Of course, there is no purchase as major as buying a car.

Consumer Reports' automobile reviews probably are their most popular.  They dedicate an entire issue of their magazine to new cars, and every other issues has several review of newly released models.  As the go-to source for many car buyers, their reviews are extremely influential.  Many people choose a vehicle based upon the CR's reviews.  A high score in their review potentially can lead to thousands of sales.  The question that I want to explore is how much stock should people put in these reviews.  In other words, I am here to review the reviewer.

On the plus side, CR has several things going for them right off the bat.  First, they have gone to great lengths to eliminate any conflicts of interest.  Their magazine doesn't accept any advertising from any third parties.  This means that they have no incentive to soften a review in order to keep a big advertiser happy.  They also buy all of their vehicles from dealerships just like you and me.  Because they don't free loaner cars from the manufacturers as gifts, they eliminate another source of conflict.  In addition, this ensures that they get actual cars that real life people are going to drive.  Many other magazines accept demo cars from the manufacturer which may be specially tuned to do well on a road test.

Second, their testing methodology is very consistent and very thorough.  They have their own testing facility where every reviewed car is put through its paces.  They subject each car to the same battery of tests so that they truly can compare the results.  Other car magazines vary their road tests from month to month, so it is not feasible to compare the results of one test to another.  Also, they testing is quite thorough.  They look at all of the aspects of a car that are of interest to most consumers, and they literally spend months behind the wheel of each test car.  That ensures that they are able to form a true opinion of how a car performs.  Other car magazines only might spend a day or two with each test car.  As many of us know, there is a big difference between how a car performs on a short test drive and in actual day-to-day usage.

All that being said, I have two bones to pick with their rating system.

1. They don't reveal their scoring methodology:

Fans of CR know that the magazine boils everything down to a numerical score.  The higher the score, the better the product according to CR.  However, they do not provide any quantitative information about how the score is derived.  Sure they mention, in broad terms, that they look at acceleration, handling, fuel economy, comfort, etc.  However, they don't reveal how these different elements factor into the score.  Do they give a maximum of 10 points for fuel economy or only 5?  Is acceleration weighted more than handling?  Why is this information good to know?  Because it gives the reader some clue as to what is important to CR.  This information is key to judging their ratings.

Cars (and all product for that matter) can be judged on a variety of criteria:  how fast does it go, how well does it handle, how comfortable are the seats, how safe is it, and so forth.CR says that they subject each vehicle to more than 50 different tests, but for every test, there are probably hundreds of other possible tests you could perform.  Different criteria might be important to different people.  If you have kids, you might value safety more than acceleration.  On the other hand, if you are young and single, you might not care about safety as much.  You might value handling more.

Every reviewer has their own weightings for all of the possible criteria on which you can judge a vehicle.  A good reviewer will tell you up front what they value so that way you know how to interpret their review.  If a reviewer primarily is looking at the performance aspects of a car, you aren't going to value their opinion if you are looking for a family vehicle.  A minivan shopper probably will trust the opinion of a review who is judging the vehicle based upon its utility and safety.

Likewise, they do not provide any guidance on how to compare scores to one another.  The Toyota Highlander has a score of 81 while the Honda Pilot's score is 74.  Both are at the high end of the Very Good range and both are recommended by CR, but the Toyota is seven points higher.  What does that mean exactly in real world terms?  Is seven points a significant gap or is the difference minor?  If a seven point difference means that the Highlander has an inch more headroom and slightly better acceleration, then it is probably a toss-up between the two and those seven points might not matter in the grand scheme of things.  On the other hand, if seven points means that the Highlander is vastly superior, then those points do matter.  However, we don't know because the scoring system is not published anywhere.

Why is this important?  Because there probably are a large segment of people who might look at the seven point difference and immediately eliminate the Pilot for consideration altogether.  After all, 81 is better than 74 and who wants to buy a car that is a 74 when you can get an 81?  However, if those seven points were gained because of minor things (or things that the buying might not even care about), then maybe the buyer is better off looking at the Pilot, too.  Maybe those seven points were earned because of the third row comfort, but the buyer doesn't care about the third row because he is going to be hauling cargo.  In that case, the gap between the two goes away.

2. The reliability rating is meaningless:

The other aspect of CR's review is the relative reliability of the car.  In order to gauge each car's reliability, they send out a survey to all of their subscribers asking them to report problems that they've experienced over the past year.  Based upon the results, they assign a reliability rating:  much worse than average, worse than average, average, better than average, much better than average.

However, what do these scores really mean?  According to CR, here is what they mean:

<1%:  Much better than average
1% - 2%:  Better than average
2% - 3%:  Average
>3%:  Worse or much worse than average

They don't reveal what the cutoff is between worse and much worse.  However, in the best case, a worse than average car has a failure rate of 4%.  That may sound "high", especially compared with the better than average categories.  However, if we flip that around and show the success rate (% of cars without a failure), here what it looks like:

>99%: Much better than average

98% - 99%: Better than average
97% - 98%: Average
<97%: Worse or much worse than average

If a model of car has no problems 96% of the time, it would end up in the worse than average category, and it would not get recommended by CR.  However, from a practical standpoint, there isn't much difference in reliability between the 96% of the worse than average vehicle and the 98% of the better than average vehicle.

Of course, that assumes that the scores are even accurate.  CR says that they only publish reliability ratings for vehicles that received at least 100 response.  That really isn't enough to get a statistically significant sampling.  Let's say that you have a particular model that has 500,000 vehicles on the road.  Of those 500,000 vehicles only 1% had a problem.  That would put the car in the better than average category which would earn it a recommendation from CR.  However, CR isn't basing their results on looking at all 500,000 vehicles on the road; they are basing on the 100 owners who returned the survey.  If you take a random sample of 100 owners out of 500,000 vehicles, there is a good chance that more than 1% of that 100 vehicle sample are going to have a problem.  There is even a good chance that more than 3% of those 100 randomly selected vehicles are going to have a problem.  CR would put that vehicle into the worse than average category based upon 100 vehicles when in truth, that vehicle should be in the much better than average category.  After all, it only takes 3 problems out of the 100 sampled to give the car a negative rating - not very statistically sound.

There are other problems with the reliability rating.  They rely upon the owner judgement in determining what is a problem and what isn't.  Unlike their road testing, there is no consistency.  Also, each person drives a car differently.  Is it fair to report something as a problem when it was caused by "user error"?  I could keep on going, but you get the point.

The bottom line is that no reviewing system is perfect, even Consumer Reports.  While they certainly are a useful data point, you should not treat it (or any reviewers' opinion) as gospel.

No comments:

Post a Comment