[wikiwawa]

A large part of statistics is devoted to making valid inference from noisy samples, largely by making sensible assumptions. Even in your example, it is common practice to assess a surgeon on outcomes of surgery, even though a lot of a surgeon's work does not lead to surgery.

We aren't claiming to assess the wide practice of law, just litigation. We do Litigation Analytics.

[selectron]

This is a great point about making sensible assumptions. Too often I see evidence that people think that data analysis should be devoid of assumptions, and any assumptions completely invalidate the analysis. In reality, almost all analysis will have some assumptions, and the good questions to ask is how plausible are the assumptions and what is the plausible impact of what happens if the assumptions are wrong.

[andrewfong]

What do you count as a "court judgment" and how are they evaluated? Are, for instance, a successful motion to dismiss and a jury decision for the defendant weighed equally? Or do you give higher weight to the motion to dismiss since that's usually a better outcome for the defendant than making it to trial?

And do you look at motions other than actual judgments? I.e. if Firm A wins a motion to suppress some piece of evidence and then there's a subsequent settlement, we should infer that Firm A "won" the case, even if there's no judgment to that effect.

[hobbyjogger]

So you sue me and get a judgment for $100,000.

If I felt that I owed you nothing, surely this counts as a "loss" for me at trial.

If, however, you had originally sued for $50M and I felt I might owe you some money but no more than $1M so I decided to fight it, the $100k judgment would be considered an enormous "win" for me.

What data does your analysis use to distinguish between the two scenarios given that the verdicts are identical?

[umanwizard]

do you understand the difference between "noisy" and "biased"?

[wikiwawa]

I understand sarcasm, that's for sure.

As a matter develops, parties tend to drop out if they think that their chances have declined beyond some threshold. But if they continue, or more likely, their legal representation suggests they continue, it implies that they think they have a good chance of success.

Of course, you can get irrational litigants but most of the time, if it goes to judgement, both parties think they have a high chance of success. Otherwise they would have bailed.

[skybrian]

It seems like surgery has similar issues. Some surgeons might only take easy cases, while others might take higher risk cases.