[tentacleuno]

I really like it when people in your position at such a big company post on here, even if it is a brief comment like this. Thank you!

[jgrahamc]

I would have posted faster but I was too busy doing my own little bit of debugging which consisted of

    1. dig @1.1.1.1 jgc.org
    2. nc -v 104.22.11.223 80
    3. curl -v https://jgc.org/cdn-cgi/trace
    4. curl -v https://jgc.org/

Hmm #1 was fast so network is routing OK. Hmm #2 was fast so TCP is OK. Hmm #3 was fast so I know (because I worked on that code) that this code path is good. Hmm #4 is slow so that means component X is slow but still working.

Of course, in parallel I'm in a conference call with about 40 other people who have actual access to monitoring and systems and other things who can see exactly where things are.

But I was damn close with four commands and gave me confidence in what people were saying. But, I have to say, Cloudflare's internal distributed tracing system is pretty cool because I got sent a trace and you could see right where the slowdown was.

[tailspin2019]

Now you're just showing off :)

[jgrahamc]

Allow an old man the fantasy that he still knows how the whole of the system works.

[maxgashkov]

That's a huge problem for companies of almost any scale. Can you shed some light on tools used internally in Cloudflare for tracing?

[jatone]

sounds like you do. ;)

[vinay_ys]

What was component X? Was it a buggy rollout?

[jgrahamc]

A proxy that serves traffic. No, it was way more complicated than “buggy rollout”.

[tuananh]

you often do you code these days?

[jgrahamc]

Hardly ever. If I start a project I don’t end up having time to finish or maintain it which isn’t fair to the team.

If I write something it’s for my own use. And I like to write things that test Cloudflare. Доверяй, но проверяй.

[maxgashkov]

I see comrades you have on your team have taught you well /jk