[craigjb]

Yah, I understand it's not always an option.

If the data size is limiting though, you can embed a Redis key or Hadoop FS reference into the message, rather than breaking your message system with massive data.

Then, you can have the storage system listen for completion messages, and cleanup appropriately. Or, like we did, just shove out old data LRU style. This way some history is always available for debugging.