[throwawaycert]

Large financial organizations have teams of analysts whose entire job is to relate consumer information, such as credit scores, to statistical outcomes. If the credit scores weren't giving them useful information, that would show up rather quickly.

[brazzledazzle]

I’m suddenly reminded of the ratings used to fuel the subprime mortgage crisis.

[throwawaycert]

That was something subtly different. The banks more or less knew they were taking poor risks, the problem is it wasn't their risk to take. They quietly packaged these bad mortgages along with others and resold the mess to less sophisticated investors (like pension plans). Then the bubble burst, the well dried up, and banks started collapsing. It wasn't a mismanagement of risk, it was criminal fraud writ large.

[mikeash]

I would argue that credit bureaus leaking all your information to criminals and then charging you money to protect you from those criminals is also criminal fraud writ large.

[throwawaycert]

That doesn't seem relevant to the question of whether credit scores are statistically useful.

[mikeash]

We've established that sufficiently advanced fraud can overcome the ability of organizations to vet the info that’s given to them. If the credit bureaus are actually organizations dedicated to fraud, then the fact that many other organization fall for it doesn’t tell us too much about their accuracy.

[throwawaycert]

If Osama bin Laden's ghost appears to you and says "There will not be a terrorist attack in New York tomorrow.", and then there is, and this repeats a few times, the fact that his statement is, facially, a lie, and the fact that he can't even exist doesn't matter. You have an input that correlates with a certain output. The intent of the person who provided you the input, or even how nonsensical you think the input is, doesn't change its statistical usefulness.

[endorphone]

A bank isn't an individual who is lending his own money. It is an organization that lends other people's money and needs a veneer of justifications for the choices they make, and credit reporting agencies are a perfect choice. You mentioned earlier that banks have statistical teams making these analyses, but most simply don't because that provides counter-information that undermines the whole value of credit reporting agencies (which is that they outsource the politically fraught choices and makes it easy to say "well we just followed their ratings, just like everyone else").

Regarding the subprime crisis, the biggest victims were the largest banks -- the most sophisticated being put completely out of business -- so not sure what the bit about pension plans comes from (many of those made a lot of money on it).