[coward123]

I spent a number of years as a traveling consultant fixing problems... I guess it was aligned with what the cool kids call Site Reliability Engineering these days. Some company would have a crisis and I'd go in and fix it. So basically I had a matter of a few hours to learn as much as I could about some huge system that had probably been built over many years by a whole lot of people.

To do this, I used a number of generally proprietary tools, where part of the project was getting the company to them buy these tools and fix their practices so that perhaps they would avoid the problem in the future.

Anyway, my point is there are open source and proprietary tools out there in the world that will instrument an application and generally based on usage patterns will build out various visualization of the application. While these are often sold or marketed as tools for operational triage, I always argued that the best practice was for developers to use these tools in order to both understand the performance characteristics of their work as well as look for unexpected interactions.

These tools will generally build out a topology of how various components interact and will often do other type of visualizations that will show class and method level chain of invocation. The good ones show end to end across distributed systems every bit of code that gets called from the time a user attempts some kind of action.

The benefit of this approach over a UML diagram or something similar is that it shows how the system was actually built and is working, rather than what a developer intended to be built. The larger the system and the more years / developers involved, the greater the delta.

[itsmemattchung]

> So basically I had a matter of a few hours to learn as much as I could about some huge system that had probably been built over many years by a whole lot of people

What are some of the open source tools that you are referring to? Within a few hours, I can imagine it being difficult to integrate and deploy your tools into an existing stack. But if you are tapping into some mirrored network port, that would make more sense.

[coward123]

Hello Matt - I deliberately didn't name any tools, as the ones I worked with are no long in the tooth and were $$$. What I wanted to do was bring to the OP some concepts that they might search for in order to identify the right tool for their purpose. To your point - yes, some of the tools I used made use of the kind of network port mirroring similar to what it appears you developed at Amazon. Others instrumented various virtual machines, others used various external monitors.