[jeffbee]

Not a big fan of this pattern. I feel like the state of the application should be maintained in the usual way, with just some struct fields or global variables. It is easy for any reader of the code to understand what is meant by a statement like `bytesReceived += msgLen`. To observe this state, you can have a function that reads and exports bytesReceived, on demand and when/if needed. This provides for the best flexibility and maintainability of the system, since you will be able to change your observability stack at a later time, or have more than one of them, without changing the stats-keeping statements at the point where they appear in the application code. This also provides the best scalability and performance since you are able to aggregate separate counters from multiple threads, if needed, minimize or optimize locking, etc.

The problem is there are a lot of off-the-shelf observability frameworks for Go that require the pattern in the article: you emit a value to a hook function at the point of production. This sucks at any reasonable scale because you are now required to just take a global lock (or write a channel, which is the same thing) every time you record a value. This is a significant contributing reason why Go servers fall apart when given access to more than a handful of CPU cores.

[gobwas]

I think all mentioned issues are related to the implementation of the user probes, not the pattern.

You mentioned an observation methods, but essentially they are absolutely the same as hooks, just inverted (with a bit less overhead on branching and hook call). E.g. your example with bytesReceived counter can be implemented with atomic operation and further export on demand by some other goroutine.

[jeffbee]

The thing I keep in mind is that state is mutated far more than it is observed. You might handle 1000 or more requests per second and only export the number of requests once per second or less. In light of this ratio it’s important to make the recording path as simple and cheap as possible, and it’s ok if the observing path has to be complicated to compensate.

I usually refuse to use atomic increments for services because it scales very poorly. Even a mutex-protected increment scales much better than atomic increment.

[gobwas]

I see your point that recording must be much cheaper than exporting, and I completely agree with it.

Anyway how will you collect that counters inside your component (to be observed later on demand)?