[matticakes]

there are a few (growing) number of production installations of NSQ...

our (bitly's) cluster spans a a few datacenters and hits peaks of 80k messages/second.

I can answer any questions you have (one of the authors)

[daemon13]

Any advice on dealing with de-duping?

Also, how was performance using JSON data format compared with ProtoBuffers & MsgPack?

[matticakes]

NSQ treats the message data as an opaque blob so the format wouldn't directly affect it (except on some lower level related to overall message size I suppose). It would impact your producers (encoding) and consumers (decoding), obviously.

re: de-duping - there are lots of things to consider, I highly recommend reading through http://cs.brown.edu/courses/csci2270/archives/2012/papers/we..., it's a fantastic paper. At a high level the answer is idempotency.

What sort of use case are you thinking of? (context would help answer your de-dupe question)