Hacker News new | ask | show | jobs
by danharaj 2232 days ago
There are hard limits to predicting the future when it involves fat tailed processes. Like pandemics, unsurprisingly. The law of large numbers gets less and less useful, for example, as you examine alpha stable distributions that have heavier tails than a gaussian. The mean even becomes undefined for extreme distributions.

Given that extreme events tend to belong to these sorts of difficult to predict distributions and have a high chance of ruin, fear is far more rational than pretending you can apply empirical methods that work for mundane phenomenon.

This is the study of extreme value theory and estimators for heavy tailed distributions. A well known popularizer of these mathematics with applications to events like pandemics is Nassim Taleb. His style is very brash and arrogant but his mastery of probability theory in this domain is very solid.

The appropriate reaction to a highly unpredictable event with potential widespread impact will look like overreaction to a naive rationalist post hoc. Don't fall into that sophistic trap. Your lizard brain knows a lot more than you give it credit for and you debase your intelligence by innapropriately elevating your rational brain.

1 comments

> "...Nassim Taleb. His style is very brash and arrogant..."

i bought black swan way back when, because it was popular and a couple people i knew recommended it. boy was that a mistake. i can overlook arrogance if it comes with insight, but that book was all incoherence and convolution, no meat. couldn't even finish, it was so bad. his fund backed by that strategy has underperformed. so appealing to taleb's authority is not a winning argument.

> "The appropriate reaction to a highly unpredictable event with potential widespread impact will look like overreaction to a naive rationalist post hoc."

the appropriate reaction to an unpredictable event is no reaction at all, since by definition, you can't predict what it is, much less when it will happen.

> "Your lizard brain knows a lot more than you give it credit for..."

your lizard brain is designed to handle futures on the order of seconds and minutes, not months and years, and certainly not decades and centuries. it's really good at predicting (and avoiding) the path of a bird for example, but not so good at predicting (and avoiding) drought.

edit: also, fat-tailed typically means a tiny bit fatter than a guassian tail, not some wavy, unknownably fat "tail". otherwise, we'd have enough known events from which to draw reasonable predictions, making it not extreme and unpredictable, nor even a tail at all.

I don't bother with his pop books and I read his tweets for amusement (and hard calculus problems). His papers and technical write-ups are worth something. Like I'm sure if he had an editor they would hate his latest book- it's god awfully written- but the mathematics is good I swear. I've studied a couple of his references. https://arxiv.org/abs/2001.10488

I don't think his fund is structured to beat the market. If his fund is structured for extraordinary events then I would expect it to underperform strategies that are optimized for typical conditions. I would expect his fund to be used as a hedge in a more diversified strategy and IIRC there was publicity that it did exactly what it was intended to do for its clients when COVID-19 became a pandemic.

> your lizard brain is designed to handle futures on the order of seconds and minutes, not months and years, and certainly not decades and centuries.

I guess I should eat my words for using an imprecise phrase like lizard brain but I'm talking about the limbic system which is most assuredly involved in decisions involving fear and ignorance and long term behavior, not snap decisions. Fear in ignorance is good if the possibility space you're ignorant of contains catastrophic consequences. If you have to make a decision that is beyond your intellectual capacity, you should be cautious. We evolved from very stupid creatures that needed to survive phenomena they did not understand. We have not transcended that need. Rationality is also knowing the limits of your rationality.

> the appropriate reaction to an unpredictable event is no reaction at all, since by definition, you can't predict what it is, much less when it will happen.

I'm sorry but this is collective suicide in the face of a pandemic. In the face of an exponentially self-propagating phenomenon that has a wide possibility of effects including: death, permanent injury and the collapse of the healthcare system, waiting around to get more information just means it would be too late if action was needed. Even if it turns out we could have gotten away without it this time, that doesn't mean we overreacted.

> edit: also, fat-tailed typically means a tiny bit fatter than a guassian tail, not some wavy, unknownably fat "tail". otherwise, we'd have enough known events from which to draw reasonable predictions, making it not extreme and unpredictable, nor even a tail at all.

Is this a philosophical point? Are you saying that if I cannot draw enough samples to approximate it that the population distribution doesn't exist?

I'm arguing that there are limits to empirical methods in the face of extraordinary events and I'm making that claim somewhat more concrete by appealing to the divergence in behavior between the population and sample statistics of heavy-tailed, e.g. something with power law decaying tails like a pareto distribution.

Is it ridiculous to conclude that in the face of extraordinary events that cannot be reasonably predicted empirically whose outcomes would be A) widespread, basically affecting everyone and B) potentially catastrophic, we have to be cautious? I don't see how.

i'll concede taleb's math might be better than his writing, but i'm not convinced his reasoning from it is sound. and yes, his fund is likely a hedge in a larger strategy, but one success is hardly indicative of predictive power. we'd need millenia of data to even get a sense of that (note that snake-oil salesmen also rely on not easily falsifiable claims).

> "Fear in ignorance is good if the possibility space you're ignorant of contains catastrophic consequences."

sure, a fear of the unknown is baked into the limbic system, but herein is the crux of our disagreement: what fear is and is not, what's catastrophic and what's not, and the magnitude of "contains" relative to preparation and response.

fear is a cascading physiological process that triggers our fight-or-flight response (lizard brain), causing physical and emotional degradation, and shutting down our rational brains. fear retards our ability to think and act appropriately. unless the threat is something imminently immediate, like an oncoming car, fear can easily overwhelm our response with misjudgment (hoarding, for example).

concern and caution--cousins of fear--are frontal, intellectual processes built atop the limbic. these, along with diversity of thought and appropriate communication, can lead to reasonable responses to non-imminent threats. this is what we should lean into, not fear.

the mistake with covid (and 9/11, etc.) is not recognizing the difference.

covid is trending toward 5% of yearly US deaths. certainly concerning and worthy of serious response, but not civilization destroying. that's the right context to couch our reactions.