[shadowsun7]

I should note that this essay kicks off an entire series that eventually culminates in a detailed examination of the Amazon Weekly Business Review (which takes some time to get to because of a) an NDA, and b) it took some time to test it in practice). The Goodhart’s Law essay uses publicly available information about the WBR to explain how to defeat Goodhart’s Law (since the ideas it draws from are five decades old); the WBR itself is a two decades-old mechanism on how to actually accomplish these high-falutin’ goals.

https://commoncog.com/the-amazon-weekly-business-review/

Over the past year, Roger and I have been talking about the difficulty of spreading these ideas. The WBR works, but as the essay shows, it is an interlocking set of processes that solves for a bunch of socio-technical problems. It is not easy to get companies to adopt such large changes.

As a companion to the essay, here is a sequence of cases about companies putting these ideas to practice:

https://commoncog.com/c/concepts/data-driven/

The common thing in all these essays is that it doesn’t stop at high-falutin’ (or conceptual) recommendation, but actually dives into real world application and practice. Yes, it’s nice to say “let’s have a re-evaluation date.” But what does it actually look like to get folks to do that at scale?

Well, the WBR is one way that works in practice, at scale, and with some success in multiple companies. And we keep finding nuances in our own practice: https://x.com/ejames_c/status/1849648179337371816

[skmurphy]

It looks like any other decision record where you set a date to evaluate the impact of a policy or course of action and make sure it's working out the way that you had anticipated.

[shadowsun7]

And how are you going to tell that when you have a) variation (that is, every metric wiggles wildly)? And also b) how are you able to tell if it has or hasn’t impacted other parts of your business if you do not have a method for uncovering the causal model of your business (like that aquarium quote you cited earlier?)

Reality has a lot of detail. It’s nice to quote books about goals. It’s a different thing entirely to achieve them in practice with a real business.

[skmurphy]

I agree that reality is complex, but I worry you are conflating the challenges of running an Amazon-scale business with running the smaller businesses that most of the entrepreneurs on HN will need to manage. I thought Roger offered a more practical approach in about 10% of the words that you took. I am sorry if I have offended you; I was trying to save the entrepreneurs on HN time.

As to Jack Stack's book, I think the genius of his approach is communicating simple decision rules to the folks on the front line instead of trying to establish a complex model at the executive level that can become more removed from day-to-day realities. In my experience, which involves working in a variety of roles in startups and multi-billion dollar businesses over the better part of five decades, simple rules updated based on your best judgment risk "extinction by instinct" but outperform the "analysis paralysis" that comes from trying to develop overly complex models.

Reasonable men may differ.

[shadowsun7]

This comment is for HN readers who might be interested in solutions.

My two questions (a) and (b) were not rhetorical. Let’s get concrete.

a) You are advising a company to “check back after a certain period”. After the certain period, they come back to you with the following graph:

https://commoncog.com/content/images/2024/01/prospect_calls_...

“How did we do? Did we improve?”

How do you answer? Notice that this is a problem regardless of whether you are a big company or a small company.

b) 3 months later, your client comes back and asks: “we are having trouble with customer support. How do we know that it’s not related to this change we made?” With your superior experience working with hundreds of startups, you are able to tell them if it is or isn’t after some investigation. Your client asks you: “how can we do that for ourselves without calling on you every time we see something weird?”

How do you answer?

(My answers are in the WBR essay and the essay that comes immediately before that, natch)

It is a common excuse to wave away these ideas with “oh, these are big company solutions, not applicable to small businesses.” But a) I have applied these ideas to my own small business and doubled revenue; also b) in 1992 Donald Wheeler applied these methods to a small Japanese night club and then wrote a whole book about the results: https://www.amazon.sg/Spc-Esquire-Club-Donald-Wheeler/dp/094...

Wheeler wanted to prove, (and I wanted to verify), that ‘tools to understand how your business ACTUALLY works’ are uniformly applicable regardless of company size.

If anyone reading this is interested in being able to answer confidently to both questions, I recommend reading my essays to start with (there’s enough in front of the paywall to be useful) and then jump straight to Wheeler. I recommend Understanding Variation, which was originally developed as a 1993 presentation to managers at DuPont (which means it is light on statistics).