[jboy55]

I've been the victim of this, a producer consumer system with a single index file using some variant of a heap. For each incoming request the producer; locked the file, read the heap from the file, added the value to the heap, then wrote it back and unlocked it. For each consumer thread, the thread did the reverse. It was setup with threads created by incoming requests, and a threadpool of 128 threads for the consumer.

This worked seamlessly until one day, at 3am PST, a flood of requests came in and the index file grew to ~4mb, and everything ground to a halt.

The legend of the architecture was a senior engineer did this on a weekend, then a month later left the company.