Scoring SCORE

In an article in their vehicles section, Forbes online published a list of the least safe 20 vehicles, ranked according to the Statisical Combination Of Risk Elements (SCORE) metric. The SCORE metric is published by an organization called Informed For Life (which I have not heard of before), and is an attempt to rate how safe a vehicle is by combining various statistics into one number.

I have some reservations about the usefulness of the SCORE metric, however, because it doesn't really correlate that well with actual driver death rates, as published by the IIHS.

Here is a plot published by IFL which claims to show the correlation between the SCORE an driver death rates per million registered years:


What you might notice right away is the large spread of SCORE values for any given driver death rate bin. For example, for the 50-60 bin, which is well below average, the SCORE varies from 80 to 180--from 20% better than average to 80% worse than average. I also notice the R^2 value of 0.48, which is not a stellar mean squared error. SCORE clearly has some correlation with driver death rates, but it is rough.

Here are some specific examples of vehicles that have high (bad) SCORE scores but have pretty good real-world safety records:

• Mid-sized SUV: 2004 Jeep Grand Cherokee
• SCORE = 181 (ave = 100)
  
• IIHS DDR = 57 (ave = 79)
  
• Small SUV: 2004 Ford Escape
• SCORE = 172
• IIHS DDR = 65
• Mid-sized car: 2004 Buick Regal
• SCORE = 123
• IIHS DDR = 57
  
• Compact car: 2004 Saturn Ion
• SCORE = 114
• IIHS DDR = 67
• Compact car: 2004 Pontiac Vibe
• SCORE = 115
• IIHS DDR = 62

All of these have higher-than average risk according to the SCORE metric, but in real life have lower than average fatality rates! It seems that SCORE puts too much emphasis on crash survivability, and not enough other factors like the ability of a vehicle to avoid an accident. It also seems to over-state the danger of rollovers. Something that is not widely publicized is that 70% of rollover fatalities occur when occupants are not wearing seatbelts (source: NHTSA). However, most people (~80%) are wearing their seatbelts nowadays, which partly accounts for why SUVs have a good safety record overall. The SCORE is really an estimate of how safe a car is in an assumed buy unspecified crash.

Aside; The IIHS driver death rate statistic also has some problems. IIHS does not account for driver behavior, which causes some vehicles such as sports cars to have much higher driver death rates than their crash test results would suggest. An interesting example of this is the DDR number for the Ford Mustang convertible (DDR 97) vs Ford Mustang hard-top (DDR 150). The two cars have nearly designs, from a crash point of view, but the Mustang hard-top has a much higher single vehicle death rate. It would appear that hard-top owners drive stupid more frequently than convertible owners.

The SCORE metric is an inconsistent predictor of vehicle safety in the real world. That doesn't mean it is worthless, but like anything, even the IIHS driver death rate statistic, it should be considered as part of a larger picture.

Autoblog Advice

Some advice to the guys over at Autoblog: slow it down, 'kay? You aren't Engadget. I know you feel like you have to stay on top of the automotive news world, but most of us can't check your site every two hours. Miss a day and you miss like two front pages worth of stuff. Raise your standards for what is worth posting and what isn't (hint: do it yourself articles by Eric are awesome, warmed over press releases are not). Otherwise, you risk burning out your readers.

LemonFree

I got an email from an outfit called LemonFree, which is an upstart auto classified service. Lemonfree's method is to offer free auto ads, and to make money from advertising. They have a nice, uncluttered web site. A very nice feature is that they offer free hosting of up to 10 photos with your ad. They don't have many ads yet, though, so if you try to actually search for a car, you will probably get a list of eBay motors auctions. Hopefully they will do better than that in the future. I'm also concerned about their name--it implies that they check the quality of the vehicles that are listed, but in fact, they do not. So it is possible to get a lemon. I wish them luck--the 800lb gorilla in this business is Craig's list.