[cduzz]

One observation / warning I have WRT msgpack is that I have seen situations where data getting serialized directly to msgpack can't be rendered as proper json. Specifically JSON requires some proper encoding of characters while msgpack is perfectly happy to convey some binary characters.

The specific situation I saw was 1) antique perl application barfs a sql dump into msgpack 2) fluentd takes that, turns it into a thing like json but with a control character in it (think old time cyan blinky happy face or \0x03) 3) that thing gets dumped into kafka 4) logstash pulls that thing out of kafka and tries to feed it to elasticsearch 5) elasticsearch reports that it doesn't like the blinky happy face, generates an extensive log, and stores the event minus the key/value pair with the blinky happy face.

Certainly many of these steps have an implicit "don't do that!" or "Update to a newer thinger!" but the root cause (perl serializer into msgpack renders a thing that's valid msgpack but not valid json) is surprising in an unpleasant way.