Hacker News new | ask | show | jobs
by obfuscurity_ 5176 days ago
I don't follow. Graphite doesn't "throw anything away". You define your own retention policy and Graphite follows it. You're free to maintain 1-second resolution for 50 years, if you have the storage capacity for it. But that isn't what this article is about.

The example covers a particular metric use-case for annotation-style details, and how to extract a meaningful aggregate from what is otherwise contextual data. Personally I love that Graphite lets me "hack about" with multiple functions. But I'm a data whore, YMMV. ;-)
1 comments
tbh 5176 days ago
Perhaps "throw away" is the wrong phrasing. The aggregates are always averages, no absolute counts. After a rollup, the maximum of some value will no longer be the maximum - it'll be the average of a bunch of maximums. There's no type information with which one can maintain this, and no observation/sum counting. For 'data whore' people like you and I, this should matter. :) (or at least, it does to me)

The article seems to be about trying to get an approximation of observation count data out of Graphite, which doesn't really store that.

link
obfuscurity_ 5176 days ago
It seems to me that you're referring to a) other data outside of the scope of this example and b) the fact that Whisper uses average data for rollups. This is a valid issue and one of the reasons for the new Ceres format.

The metrics I demonstrate in the article do not rollup. If they did I'd completely lose the data. That's why I keep them at 1-sec resolution for their lifetime (generally 1 year). In hindsight it seems your original comment was a general critique of one aspect of Graphite and outside the scope of this particular article.

link