[nathanielc]

Several years ago and I don't remember much of the specifics but we had an issue with static content being served from our site being randomly truncated (polluting the cache etc).

We eventually traced the issue down to the Nginx server that was serving the files and one of it's cache buffer size config options, (I don't remember which one anymore). We noticed if the file being served was larger than a certain size it would occasionally truncate the file but not always. We tested increasing the buffer size by repeatedly doubling the default value, which was a power of two, up to a size of several GBs. But the files kept being truncated for some small percentage of the requests. At this point we knew it wasn't directly related to the size of the buffer since it was larger than any files being served. Finally someone suggested that we test a value that wasn't a power of two and the issue was gone.

We figured it was an internal bug in Nginx where it was growing an allocation buffer and used powers of two, but had an off by one error that didn't copy the second half of the buffer or something. We dug through the code but never found anything and so we left the cache setting at +1 from the default power of two value and never had an issue again.