[pinkmuffinere]

> But let’s think about that 91% failure rate for a moment. When I bring this up in presentations, I invite the audience to consider what the auto industry would look like of 91% of new car designs proved unable to roll out of the factory, or if 91% of new airliner models were unable to leave the ground - and if you only found that out after spending all the R&D money to build them at full size and trying to fly them. No cutting-edge restaurant could survive if 91% of its innovative dishes proved inedible or outright poisonous. What other industries operate under these bizarre conditions?

This is bizarre, bordering on stupid. Frist of all, there likely is ~90% failure rate of prototypes; I feel that roughly matches my experience in engineering. Of course the design that makes it through the process, testing, refinement, and into mass production is not going to have a 90% failure rate, but that's a _finished product_, whereas clinical tests are just that -- tests. Finished cars are more analogous to individual pills coming out of the factory. And I'm not even sure the analogy would be very meaningful anyways, because we have different requirements for things of different impact and importance. A 10% manufacturing defect rate is fine in forks, but not for fire extinguishers.

[pama]

The drug candidates that enter human clinical trials, (phase 1 to phase 3) are identical in all ways to the final product if approved. The company is not allowed to change any part of the manufacturing process. All the research for how to make the drug is done in preclinical phases. In the clinical phase we test the unknown human biology with a final product candidate.

[fabian2k]

A large part of that failure rate is in phase 3. Your comparison with prototypes would be more like drug development before even phase 1. Phase 3 is enormously expensive, much more than the earlier parts. And what's even worse, you don't get to learn all that much from failures in drug development in many cases. You can't just fix the problem and try again, you essentially have to try a completely new molecule.

[pinkmuffinere]

That's fair, phase 3 testing isn't an early prototype, it's a prototype with many years of effort behind it. But I still think it is _more_ like a prototype than it is like a product coming off an assembly line. Perhaps phase 3 is like the latest prototype from boom (the company trying to bring back super-sonic jets). Those prototypes represent 100s of engineering-years put together, and are not much like the prototypes you'd encounter in consumer product development. I still think 90% failure rate is believable for that. How many significant design iterations has Boom gone through so far? I don't really know, but I could believe it's been 10 significant iterations.

[ethbr1]

Phase I testing is ~$5M. Phase II $6-20M. Phase III is $10-50M. https://pubmed.ncbi.nlm.nih.gov/26908540/ (up to 2012 numbers, non-inflation-adjusted)

Those costs are strictly for the mandatory testing, not the precursor drug design, etc.

The cost and calendar time required to be sunk before 10% pay off is why you generally only have drug development done by (a) some of the largest corporations in the world or (b) fly-by-night companies hoping to sell their preliminary drugs/results to the former to complete testing.

[pinkmuffinere]

tl;dr -- Boom's iterations are very similar to a phase III trial! Please prove me wrong!

Just to clarify, I think our numbers are in agreement here. From your numbers, phase I + II + III would be in the range $21-75M, all told. Let's double that to account for inflation (2012 is quite stale), to get $42-150M. In California, I'm going to estimate a "normal" engineer's compensation is $150-400k, and the company actually pays more than this because of benefits, HR, etc -- call it $250-600k. Let's say it's 350k fully loaded on average. Then the $42-150M number represents 120-430 engineer years. I'd totally believe that Boom has 120-400 engineers working full time, so if they make a major revision every year, then each major revision is like a full phase I-III progression!

I think there's some reverse-dunning-kruger going on in this thread, where because we're all engineers we think engineering is comparably easy. My claim is that it's actually pretty similar to medical testing, in terms of generic human effort (to the extent that can even be measured and quantified).

edit: According to wikipedia, Boom had 150 employees in 2021 [1]! I'm very proud of my estimate lol, it seems like it was right on the nose. Somebody give me a prize!

[1] https://en.wikipedia.org/wiki/Boom_Technology

[ethbr1]

The difference is that phased safety/efficacy trial costs are sunk.

If it fails, you've produced near $0 value, aside from some more information (maybe) about how something doesn't work. Because those costs are strictly for the execution of the trial, not for the development work before/during/after it.

That's in contrast to your hardware prototyping example, where a failed prototype still usually represents >$0 development value.

The appropriate analogy would be if every time a hardware prototype failed testing, it was required by regulation to scrap it, go back to the design stage, and restart from there.

[mft_]

> A large part of that failure rate is in phase 3.

I must ask for some data to support this statement, and also your definition of "a large part".

[hibikir]

It's not that much better in plant breeding: You'll find hundreds upon hundreds of supposedly better varieties tried, and 5 years later, you are lucky to get 1 good one that one would consider commercializing. And that's after years of testing changes in smaller settings. We just don't have models that are predictive enough without planting, and waiting for plants to grow takes time.

[rwmj]

Way more than hundreds! Page about finding new varieties of roses: https://eu.davidaustinroses.com/pages/breeding-programme

[addaon]

(This is your annual reminder that atomic gardening is a thing. https://en.wikipedia.org/wiki/Atomic_gardening)

[ip26]

Phase 3 seems closer to doing a soft production launch and finding customers completely reject it despite early focus groups liking it.

Prototypes, when they fail, are also often tweaked into something that works. A lot of phase 3 failures appear to have been complete dead-ends.

[lolc]

Yeah why is 10% success bad? What is the tradeoff between spent effort and missed cures if we try to tweak the success rate by killing prototypes earlier in the pipeline? While reading I was expecting a reasoned argument... that was stubbornly not coming around.

[braiamp]

It's mostly that we have too many false findings pre-clinical trials, most drug targets validated in models that don't actually hold up in humans, so a huge share of the 90% failure is money and years spent testing candidates that were never going to work. That inflates the number of potential candidates that are likely wrong, due how we select them.

There are ways used right now that are working towards reducing those false findings by looking at actual humans, their biomarkers and whenever or not there's an associated molecule to the condition that we would like to pass onto others. It will not kill prototypes, instead it will discourage us from going through a prototype at all by going for better candidates instead.

[rcxdude]

The prototypes are generally cheap though: you're not standing up a full production line for them before you find out that they're duds.

[DoctorOetker]

The problem is that you model the filtering step as a monolithic step.

Things start with academic literature, and models (qualitative or even numerically quantitative, human comprehensible, or computationally predicted effects, ...). Then academic level testing occurs, resulting in putative results, obtained on animal models, cell / tissue cultures, ... before proceeding to human trials. The failure rates are for this final step. Imagine being in control of some pharma fund, there is a huge stream of putative drugs emerging in the literature, and only limited budget AND limited test-patient slots. On average people in this position succeed in selecting one that performs as predicted only ~10% of the time. Thats not a simple exploration-exploitation trade-off. With so many candidates, assuming proper pre-human experiments, one would expect much better results, and you'd from a financial perspective redirect focus towards those drugs with high confidence from prior forms of non-human testing. Yet we see failure rates 80-90%! From a purely financial perspective, there is a huge incentive to place more selection emphasis on confidence, but either its not happening or institutions (public or private) are systematically dropping the ball.

To make the engineering analogy with design methodology: before testing a new implementation, we have the luxury to preselect implementations depending on their unit tests, considerably improving the success rate for the higher level implementation. Yet for the analogy in drug selection we fail miserably.

I don't believe the failure is proper to the selection process. Its that the true fitness function is unavailable, if we had it would simply be a matter of performing gradient descent.

But obviously earthly biology does not come with a reference manual of all niches, and analytic objective differentiable fitness functions.

To a large extent it is in fact still the exploration-exploitation curve, but a meta level.

We can't bypass natural selection, we can't turbocharge natural selection (like the Nazi's tried), not only because it is evil, but because the fitness statistic is for all purposes and intents, emergent in nature.

The average reader here will be very familiar with the concept of premature optimization: don't start micro optimizing your code in assembler before functional correctness, first go for correctness, then reason about the hot paths from profiling and investigate and optimize from there.

Historically, hospitals and medicine long predate modern science and biology.

It predates the discovery of natural selection, it predates the measurement of selection phenomena on shortlived organisms.

Healthcare is high inertia never-to-seldom-recognize-earlier-mistakes domain.

The justification of healthcare is never supported by some axiomatized formally verifiable system of ethics, it is justified on the basis of associations and vague nebulous historically grown rules.

Socialized healthcare is a world-widely supported doctrine. Don't wish unto others what you wouldn't wish onto yourself is another. There is some nebulous concept of a right to healthcare, but a right to what: some nebulous default "healthy" state? There is also a strong connection with egalitarianism, if someone gets sick from a flu we somehow believe it is desirable to help them overcome it, because we claim this flu could have afflicted any individual equally.

THe healthcare zooko's triangle looks like this, you can't have all 3 of the following, so once you have unconditionally selected one desideratum, you will be left torn by the decision between only one of the remaining desiderata:

1. egalitarian access to healthcare 2. working healthcare solutions towards "individual health", i.e. improving procreation rates statistically 3. maintained fitness of the collective human genome distribution

Assuming without compromise 1: egalitarian access to healthcare: ===========

To the extent a healthcare instrument (drugs, or tools like glasses) works (2), it undoes decreased genetic procreation rates, inducing a higher incidence rate in future generations, reducing the fitness of the human genome (3): we have not cured or treated this individual, we have traded innate health, fitness and quality of life of future generations for the convenience and comfort of individuals in the current generations.

To the extend we insist to maintain genetic fitness, it would require the healthcare instrument to be ineffective and thus not influence procreation statistics of fertile individuals. To the extent a healthcare instrument doesn't affect the procreation statistics of a fertile individual, the healthcare instrument didn't improve quality of life for this individual (and is effectively a quack measure). For example without the flu shot, an fertile individual might have met a potential mate, or been in the mood to mate with an already associated partner, but an individual without the flu shot might have felt too sick, or been to repulsive for a mate or partner during sickness. Natural selection first and foremost is about procreation probabilities and rates, and much rarer the individually stronger but collectively weaker signal of death.

For example: there is wide consensus that pre-modern tribal hunter/gatherer humans only had sub 5% incidence rates of poor vision requiring corrective glasses according to modern standards. The selective pressure on eyesight was so strong that even 1 or 2 generations of modern medicine, results in large majority of population requiring prescription glasses, the loss of a strong selective pressure quickly results in drift away from fitness.

Assuming without compromise 2. "individually effective" healthcare instruments: =============

If we select to keep (1) egalitarian access then it will be at the expense of (3) maintained fitness in the humanities collective genome: the egalitarian access to the individually effective healthcare measure, will result in the loss of genetic utility, since the utility is supported and provided externally instead of innately.

If we select to maintain the collective genetic fitness of humanity (3), it can be attained while maintaining "individually effective healthcare" measures (drugs, crutches, pacemakers,...), but only if we relax (1) egalitarian access to healthcare: a modern nation state can admit migration or otherwise support the transfer of genetic material from regions with no or little healthcare, or from regions where such healthcare was only recently introduced. Steady state reliance on such a stream of wild-type humans, is basically bio-colonialism: it requires a region where humans face natural selection without help from healthcare, and their genes are used to improve or maintain fitness of modern nation states elsewhere.

Assuming without compromise (3) maintaining fitness of humanities collective genome:

If we insist on maintaining (1) egalitarian access; then it will be at the cost of (2) healthcare that objectively improves the quality of life and thus procreation statistics in the case of fertile individuals. So we could all enjoy egalitarian access to non-functional medicine.

If we insist on maintaining (2) individually effective healthcare measures, which improve quality of life and hence procreation statistics of fertile individuals, we have to sacrifice (1) egalitarian access: if a sufficient collection of humans is basically deprived of healthcare access, then yes we could maintain fitness by selecting their genetics for procreation.

It's a veritable zooko's triangle, and an absolute nightmare once comprehension sinks in.

If for every QAPR (quality adjusted procreation rate) were taken into account for healthcare interventions, they'd all go negative! whatever current "meritocratic assessment" of healthcare instruments is basically fraud from the perspective of genetics and natural selection, like those video's you see from parents climbing school walls in India to help their children cheat on some national level exams, and somehow this being normalized by society...

It is not a question of allegiance with or against modern healthcare, it is a question of internal consistency of what is known about natural selection, selection pressure, healthcare, etc. We only know how to trade with loss (but you don't need a degree in medicine to attain that skill).

Modern medicine stems from a premature optimization objective (and the original goals did not involve (1) egalitarian access -for nobility and their armies- nor did the original goals involve (3) maintaining genetic fitness for future generations -the statistics of natural selection and concepts like selection pressure were only poorly understood).

Socialized healthcare institutions continued their "mission goals" without establishing an existence result first, they basically tried to fulfill and democratize the nebulous informal desiderata the earlier forms of healthcare aspired to... It's historically grown holy-grail level provable unobtainium. Its a pointless crusade worse than fighting windmills.