[georgeburdell]

Mine was an automated file transfer system that had to be 100% reliable on an insanely unreliable network (~95% uptime). Took about 9 months of bug squashing after development was done. So many edge cases. I would probably never mention this in a job interview because I doubt most people would understand why it was so hard.

[finaard]

Reminds me of a project from two decades ago which also had a somewhat tricky part.

We needed to do a nightly transfer of data. We had a variable amount of data to transfer, but typically in the range of one to two TB. We had a 1GBit link between the data centres housing the two systems, but it wasn't an exclusive link - backups and other stuff would be running during the night as well, so if we hog all bandwidth we'd have to deal with unhappy people. Hard deadline for the transfer was start of the work day.

Now the data does compress easily - but the data is only available for compression at the beginning of our sync window. We definitely need to compress some of the data to sync everything in time, and keep other users of the line happy. But: If we spend too much time on the compressing we might not have enough time left to send the data, plus we're not alone on the systems - other people will be unhappy about their nightly jobs failing if we hog all the available CPU time.

So we needed to find the right balance of data compression and bandwidth utilisation, taking into account all those factors, to make things work in the amount of time we had available.

Thanks to AMD nowadays we'd just throw more CPUs at the problem, but back then the 8 CPU server we were using was already quite expensive.

[bjoli]

I once wrote an inliner. When you have not done it, it seems simple. When you are doing it it is like trying to restrain a large rabid dog with a slippery leash.

Now, I am not a programmer by trade, but I have a hard time thinking anyone would find it nice to write an inliner. At least not if you want the inliner to always make things faster.

[throwaway2037]

    > an insanely unreliable network (~95% uptime)

This is wild! Can you explain more?

Did you ever blog about this program? It sounds very interesting, and there is no job interview on HN!

[georgeburdell]

Lots of things are baked into that 95% number. Sometimes the power would go out. Data recipients were careless about checking whether their computers were powered on and had a static IP assigned, but I’d certainly hear about it if they didn’t get their data. Network engineers would futz with settings and do poorly announced infrastructure upgrades. Rather than try to fix all these issues, I just wrote the code assuming we were working from a 3rd world country.

[greazy]

I'm working on this exact same thing. Was your code ever published or did you blog about it?

[georgeburdell]

Nope, just some internal facing code. The challenges boiled down to tracking what parts of the data got successfully sent to which recipients and how to get proof the data were transmitted correctly

I wish I would have found a 3rd party tool to do all this, but I never did

[adammarples]

Is this not basically just torrenting?

[georgeburdell]

I guess, but some of the recipients are just 3rd party automated systems with a standard set of instructions, so I have no control over the protocol used