[willcodeforfoo]

Somewhat related: can anyone recommend a simple solution to share each node’s ephemeral disk/“emptyDir” across the cluster? Speed is more important than durability, this is just for a temporary batch job cluster. It’d be ideal if I could stripe across nodes and expose one big volume to all pods (JBOD style)

[noctarius]

You need some type of cluster-wide shared memory?

[willcodeforfoo]

Yep! But I don't have access to any block devices on nodes, only the local paths so I'm not sure OpenEBS or Ceph would work...

[noctarius]

I guess your biggest issue may be the multiple writer problem, but you'd have the same issue on a local disk. The second multiple writer are supposed to update the same files, you'll run into issues.

Have you thought about TCP sockets between the apps and sharing state, or something like a redis database?

[willcodeforfoo]

Hmm, maybe... although that wouldn't help with the aggregating multiple nodes issue, or would require a lot of app-side logic.

In this example, I have 200GB of ephemeral storage available on each node, ideally I'd like something like this:

  node1: /tmp/data1 (200GB free space)
  node2: /tmp/data2 (200GB free space)
  node3: /tmp/data3 (200GB free space)
  node4: /tmp/data4 (200GB free space)
  node5: /tmp/data5 (200GB free space)

...pods could some how mount node{1..5} as a volume, which would have 5 * 200GB ~1TB of space to write to... multiple pods could mount it and read the same data.