[johnb]

It's funny how disk abstractions get you every time.

We used to store and process all of our uploads from our rails app on a GFS partition. GFS behaved like a normal disk most of the time, but we started having trouble processing concurrent uploads and couldn't replicate in dev.

It turned out so GFS could work at all, it had different locking than regular disks. Every time you created a new file it had to lock the containing folder. We solved it by splitting our upload folder in 1000 sequential buckets and wrote each upload to the next folder along... but it took us a long time to stop assuming it was a regular disk.

[sciurus]

FWIW, this behavior is explained early on in the documentation for GFS2.

[johnb]

As we were using EngineYard for hosting at the time, everything was set up for us and we never thought to look it up.

We now pay a lot more attention to underlying stack. Just because you've outsourced hosting (either cloud or managed physical servers), you really need to know every component yourself.