[Acconut]

Splitting a resource into multiple parts and embedding in into a single message in order to avoid external storage and the need for locking is an interesting approach, but sadly not an option for me. Nevertheless, I appreciate your response.

[craigjb]

Yah, I understand it's not always an option.

If the data size is limiting though, you can embed a Redis key or Hadoop FS reference into the message, rather than breaking your message system with massive data.

Then, you can have the storage system listen for completion messages, and cleanup appropriately. Or, like we did, just shove out old data LRU style. This way some history is always available for debugging.