[advael]

This is a good evaluation of the way risk posture can inform design decisions, but I think it sort of ignores the elephant in the room: A short term strategy wins more on average in a competitive context, especially when existential risk is on the table for entities who lose a competition. Most solutions to this problem involve solving hard coordination problems to change the balance of incentives or lower the lethality of competition. Figuring out systems that work for your level of risk tolerance is very achievable and can be incrementally improved, but designing for robustness is something that needs fractal alignment at higher meta-layers of incentives to be sustainable

[nonrandomstring]

> especially when existential risk is on the table for entities who lose a competition.

This really just explains externalities. Competition creates apparent short-range existential risk to a business. Real existential risk (to people or things damaged by your defective product because of "precarious perfection") land somewhere else - usually where their impact is much larger.

[advael]

If something is considered an externality by some system, decisions that prioritize it are by definition hard or impossible to make within that system. In other words, to change that, we have to rewrite systemic priorities to some degree

[nonrandomstring]

That's an interesting thought I'll give some chin stroking to. Sounds like something Kurt Godel and John Gall would have hatched together. Is it a personal insight, or from experience, or something you think might be attributable. cheers.

> we have to rewrite systemic priorities to some degree

maybe Forrester or Meadows has an insight?

[advael]

I mean I am sure I'm not the first person to have this insight - as I said I view it as baked into the definition of "externality" - but I'm also not pulling it from any specific source I can consciously recall

[6LLvveMx2koXfwn]

> but designing for robustness is something that needs fractal alignment at higher meta-layers of incentives to be sustainable

This is why I remain an engineer and nothing more. If this sentence ever became meaningful I guess I would have been over-promoted.

[advael]

lol okay I'll admit upon reading it again that that was a more verbose way to make that point than it needed to be

How about "prioritizing robustness and other kinds of long-termism need to be at least somewhat protected by the system of incentives they operate in to succeed"? I invoked fractal self-similarity there because I think you best protect long-termism by prioritizing long-termism in decision-making at scopes further out