[nradov]

The trouble with that is it doesn't establish causality. Did the patient's symptoms improve because of the provider's intervention or in spite of it? Plus you can't force patients to respond to the survey so there's no way to know if you have a representative sample. In my subjective experience, patients that are happy and improving are more likely to answer those questions than patients who are unhappy and ill.

[Retric]

X people per hour for over a few years averages out much random noise. Even surgeons see on average multiple people per day over a few years which is still recent enough to be relevant.

[bigfudge]

The real issue here is the selection bias in the caseloads of good vs poor physicians. In psychological treatment teams it's common for the caseload to be (implicitly or explicitly) allocated based on the perceived skill of team members. Note — this may not necessarily correlate with their actual skill, but it still screws up any estimates of provider performance unless you have very good prognostic indicators of outcome from before treatment (and you likely won't).

[nradov]

No it doesn't average out at all due to persistent differences in patient populations across providers which can't be adequately controlled for using the available data.

[Retric]

You misunderstand. You have a representative sample of the patients seen by that doctor. Individual differences like patient weight may be very important at the individual level, but across a thousands of people that's far less important than the overall differences across populations.

[nradov]

No that's simply not how it works and won't give you an accurate picture of quality differences between providers. I don't know how to make it any more clear.

[Retric]

Comparisons between providers is a separate issue.

Suppose you assigned people randomly to 101 doctors from 2 populations (A,B). Now suppose A was 10x as likely to die. D(0) get's 0% of A's and 100% B's. D(1) is 1A and 99B. All the way to D(100) that only get's B's.

In that admittedly simplified example you could determine that D(0) did a better job than D(100) by only getting 8x as many deaths even if 9.8x may be statistically irrelevant.

Yes, the real world is vastly more complex. But, while that may make a strict ordering impossible you can likely find out the best doctor is likely in the top quarter and the worst doctor very likely in the bottom quarter, which can be useful.

Picture a score card that said 80% chance in (0% - 20%], 15% chance in the (25%-50%], etc. That's not exactly meaningless information.

[nradov]

Nope that doesn't work because there's still no way to reliably control for the confounding factors. We don't even know what all the confounding factors are. Patients aren't randomly assigned to providers.