[savanaly]

Doesn't your whole post rely on the assumption that outcomes in the cases they measured don't correlate closely with outcomes in the (far bigger array of) cases they didn't measure? Don't you have to show that before you can definitely say their analysis is completely invalid?

[wahern]

Depends on what is meant by bullshit. Yes, if it meant that the study was absolutely wrong. No, if it meant that the study failed to provide any evidence whatsoever that there exists a correlation.

In science guess who has the burden?

In a legal context the burden of proof is broken down into the burden of production and the burden of persuasion. If you fail to meet the burden of production it's fair to say a claim is bullsh*t. It necessarily follows that if you haven't met the burden of production you cannot have met the burden of persuasion as a logical matter.

Remember, any claim must stand on its own terms. It doesn't matter what is the reality or truth of the matter. We can never know the absolute truth of something. We can only attain a slightly firmer grip on reality when people propose persuasive arguments. Maybe you can mine a failed argument for useful bits, but a good rule of thumb is that it's not worth the time if the argument cannot even meet a minimal burden. If it wasn't worth the claimant's time why would it be worth yours? That's strong circumstantial evidence to move on.

[libertymcateer]

Yes, it does, and yes, that is correct.

Lawyers can drop clients with cases that they think are losers and refuse to take them to trial. And, very, very often, before or directly after closing arguments, if the writing is on the wall, a case settles.

Additionally, what sorts of cases are we talking about? Insurance defense? Toxic torts? Data breach?

The 'winnability' of these cases varies highly. There are many, many law firms who's entire business model is built entirely only on taking no-brainer winning cases. Think injury lawyers. They may - perversely - have very low winning percentages, because 90-95% of their cases settle, and only the real squeakers get to trial.

Let's look at the inverse. Do you really need Quinn Emmanuel or Gibson Dunn if you have a slam-dunk case? Or do you need the best litigators around when your case is a total coin-toss? And, in that event, is it an example of bad lawyering if your crack-team loses because they are pushing the bounds of appellate advocacy?

The idea of judging 'law firms' without further taxonomic distinction, generally, by trial disposition is just - it is frankly idiotic.

[selectron]

It is always easy to criticize a data-driven analysis by saying its assumptions could be wrong. In the real world, all analysis is based on assumptions, some of which you can always claim might not be correct. But you have to really present an argument as to why and by how much the assumption is likely to be wrong, you can't just state that the assumption might be wrong. The assumption that cases which don't settle are not at all indicative of how well a lawyer performs is a very bold claim, much bolder in my mind (admittedly I am not a lawyer) than the claim that there should be some correlation between lawyer performance and results in cases that did not settle. It is possible that lower ranking lawyers settle less often, I'd like to see the data on that.

Furthermore, on average the effects you are mentioning will wash-out, unless there is a systematic bias whereby lower ranking firms and higher ranking firms settle in different manners.

[CPLX]

There's a difference between saying ones assumptions are wrong and stating that the units of measure are completely meaningless.

[jessaustin]

Why should we expect a correlation? Why should we assume they even thought about the issue, if it isn't mentioned in their analysis?

[selectron]

Why should we expect there isn't a correlation? They do mention it in their analysis. At the end of the day, just because there might be a systematic bias in your result doesn't mean there is a systematic bias.

All real-world analysis (especially for observational studies) rests on certain assumptions. It is always true that these assumptions might be wrong, but it is important to think about whether or not the assumption is plausible. It seems plausible that on average, a lawyer who is able to get better outcomes when they don't settle is also able to get better outcomes when they do settle.

Furthermore, even if most cases are settled, the rare cases that do go to trial can have an outsized impact. Usually people settle because a bad judgment is devastating (as well as not wanting to pay legal costs).