[brendangregg]

It would be great if all Linux distributions had SystemTap out-of-the-box like RHEL. Or something with a sufficient feature set to solve problems (which may become ktap, or perf_events, or lttng). However, most of the Linux performance issues I'm debugging are on Ubuntu systems. So the question remains: when will Linux-at-large get this capability?

Red Hat can certainly say that SystemTap is obviously a priority on their OS, and they can only make best effort for others. I said this in the talk. Indeed, a working SystemTap should be an incentive to run RHEL. It is a compelling argument -- the benefits of a working SystemTap are enormous (which I also explained in the talk). Sun made basically the same argument for switching to Solaris: a working DTrace.

Although, I'm not sure that using SystemTap is entirely safe for production yet, which I said was the most important priority. I notice that this related kernel panic is still not fixed:

http://sourceware.org/bugzilla/show_bug.cgi?id=2725

Although some recent and promising progress has happened in the past few months, as noted by the bug! I only care about this bug because I sometimes trace large subsets of the kernel (eg, all functions in a module or driver), and I think I've hit the same related issue. I don't actually need to trace everything.

You said that DTrace can trace from userspace to kernel, which SystemTap can't. I'm not sure if there's a specific case you're referring to, but SystemTap nowadays can indeed trace userspace to kernel (via uprobes), and a lot of work has been happening to get SystemTap userspace tapsets to work.

As for setting up SystemTap being difficult on distros that don't support or maintain SystemTap. Well, almost all of my customers run Ubuntu. If they hit perf issues, do I convince them to switch to RHEL? (The approach that Sun made for DTrace.) Or is this a problem to be taking up with the other Linux distro maintainers? Which goes back to my original point: when will Linux get this? :-)

[ema_rocca]

Is it problematic to install SystemTap on Ubuntu? On Debian systems it's straightforward.

apt-get install systemtap linux-image-amd64-dbg

[_delirium]

It's not too much harder, but requires a non-default respository, so mildly harder. Quasi-official instructions here: https://wiki.ubuntu.com/Kernel/Systemtap

The difference is that, as of the past few releases, Ubuntu doesn't ship a -dbg kernel package in the main repository like Debian does. Instead they've come up with a new system to ship 'debug symbol' variants of all packages, with a -dbgsym suffix. But they ship them in a separate repository that isn't enabled by default, I guess to avoid confusing end-users (ddebs.ubuntu.com). So you need to configure that repository, install its key, then pull linux-image-$(uname -r)-dbgsym. I guess you might also want to pull the debug-symbol versions of the main libraries as well.