[thomasknowles]

strace is the Daddy of debugging especially when you can attach it to an already existing program using it's PID. I have solved so many permissions errors problems with strace where the application itself just failed without logging anything.

[nz]

Similar experience here, except that I've solved similar issues (and far more complicated issues) with DTrace instead of strace. The range of information that DTrace can retrieve about a running application surpasses anything that strace can retrieve. DTrace can retrieve information on a systemic scale, whereas each strace instance operates on a single process. And the overhead is significantly lower with DTrace.

For many of the systems applications I design for Illumos, I've used DTrace probes as a way of logging very frequent events, on demand (to avoid frequent IO). All of the events that _must_ be logged for the application to function properly, are logged and fsync'd.

I think that in most systems dynamic tracing will eventually replace a significant portion of the logging functionality that people code into their applications.

Either way, if you think strace is sweet, give DTrace a spin.

Some helpful links:

[0] A video demostration of DTrace, by the creator of DTrace.

[0] http://www.youtube.com/watch?v=6chLw2aodYQ

[1] A post I wrote, demonstrating DTrace. Similar posts can be found on the blog's dtrace-addict page.

[1] http://nickziv.wordpress.com/2011/04/08/adventures-of-a-dtra...

[2] A wiki that contains DTrace examples for various languages and system facilities. Some examples may be Illumos-centric.

[2] http://www.solarisinternals.com/wiki/index.php/DTrace_Topics

[3] DTrace's home. Contains blogs by the engineers behind DTrace.

[3] http://www.dtrace.org

UPDATE: Meant to reply to parent's parent.

[tangothedog]

systemtap is another utility in this vein that's worth checking out

http://sourceware.org/systemtap/

[nz]

I already checked it out and the timer probes don't work at all.

On one system, I couldn't even invoke kernel functions.

On another system, the kernel panicked as soon as I executed `stap -e ...`.

Basically unusable, at this point.

I hope it improves because developers on linux could really benefit from dynamic tracing.

[nikcub]

and on OS X you have DTrace (from Solaris), which can be called using dtruss, which is a shell script wrapper around DTrace

also, in developer > applications there is an app called 'Instruments' which is an excellent GUI front-end to DTrace

[Meai]

If the application fails, wouldn't it just shut down and the process ends? Meaning that I can't call strace anymore? Do you have suggestions for that usecase? How would I know what the process id is going to be before I start the program?

[tedunangst]

$ strace program args

[zobzu]

try austrace. also gdb of course.