[paulgb]

> Any external API that could reasonably take a long time to return should have an asynchronous API.

I agree, but that doesn’t solve the problem here. The remote API was asynchronous, I was just waiting for it to ack my instruction. Because of issues out of my control (network congestion, maybe?) the time to get a 200 OK from the server shot up.

> always provide your own timeouts to what you consider reasonable for connections and responses

Agreed here as well, and I was providing my own timeout. The problem is that (cost-wise) it’s fine for 1% of requests to hit a 5-second timeout, but gets expensive when 100% of requests do. And lowering the timeout means that during normal times, requests that have latency in the tail of the distribution but ultimately go through would fail, which is undesirable.

[riquito]

I don't think that's what they meant by asynchronous api. More likely something along the lines of "schedule a job - 1 fast request" "call me back at my endpoint when the job is finished" (or variations, the gist is you are not stuck waiting for the server response)

[colinmhayes]

How can you schedule a job within 100ms if it takes 3000ms to establish a connection with the server running the other api due to network congestion? Maybe you can send the request in UDP, but you'll get tons of situations where the secondary api never ran.

[paulgb]

Right, but even in schedule a job - 1 fast request, there's still some waiting for the server to ack that it got the message. That's the request that became slow. I don't see how there's any getting around that, async API or not.