[askldjd]

I actually did look at the TCP layer early on. However, I didn't pay close attention to the TS Val. From the packet dumps, it just appeared that the TCP window had stopped sliding. I couldn't conclude that NSOC's router was at fault.

Getting NSOC on-board is a big deal. After all, they deal with the entire VA network with 100,000+ employees. If you think about it from their perspective, why is USDS' TCP connections so special?

[kevin_nisbet]

Network level troubleshooting is incredibly difficult, especially for individuals who don't have a networking background. Even showing someone how to read wireshark often isn't enough.

I just wanted to politely point out though, in this case, I think there should have been an indications of a network failure in this analysis early on, from the standpoint that TCP frames were sent to the server which were not acknowledged. This would depend on the point where you capture the traffic naturally, but the lack of acknowledgement would be a strong indicator that traffic is not reaching the server, or that replies are not reaching your capture point.

So while the TS Val may be the cause of the drops, I think the packet drops should have stood out when seeing the traffic being black holed, and likely the same segments getting re-transmitted continuously.

And for anyone out their who thinks this is easy to catch, I'd say this is very easy to miss, because you need to have a good understanding of how TCP works in the first place, to know what not working looks like.

[dboreham]

True, but Wireshark will highlight dodgy TCP frames (retransmits, dups, etc) which should give a small clue to look further. I agree that it is necessary to understand how TCP works (or have access to someone who does) in order to run Internet services.

[js2]

I applaud you for not putting the obvious work-around in place:

- Inserting "sleep 300" into the startup process.

- Adding a cronjob to reboot the servers once a week.

I kid, but I'm sure we've all seen hacks like this.