|
|
|
|
|
|
by jake_morrison
1134 days ago
|
|
|
A customer needed help with the size of their DataDog bill. A big part of that was log messages. They had poor logging discipline, so they started by filtering out all logs that were INFO or lower priority. But then the developers couldn't see what was happening in production, and they started logging messages at WARNING and ERROR in production. That made the output of unit tests in CI light up like a Christmas tree, and the output was unusable. Good times. The solution was to carefully audit all log messages, reducing the number of messages, making sure they were at the right level, and replacing log messages with trace attributes. We also implemented trace sampling to avoid paying for redundant success messages. That reduced the bill by 80%, saving tens of thousands of dollars a month. |
|
|