[akkartik]

I'm constantly finding new places in my program to add prints to. This isn't just copy changes. (Though that has also had a huge impact occasionally in understanding something. Imagine emitting 2 variables and then focusing on 1 of them. A pattern can pop out of a screenful of iterations of a loop when things line up just right.)

[aidenn0]

There are debuggers that will let you halt your program, go back in time, and then print out each place where a specific variable changes. If that's "interacting with your program through a pane of glass" then shrug.

[akkartik]

I'm very happy to hear it! Can you point me at a few of them?

[aidenn0]

Mentioned in TFA is "rr"; it serves up to gdb the history of the program and you can use tracepoints (also mentioned in TFA) to essentially retroactively add prints. Gdb is ... not particularly ergonomic, but it is eminently scriptable and writing programs to generate arbitrary logs post-facto is a useful skill.

Undo is another one for Linux programs, and I think I've seen developers from them post here.

[mark_undoio]

Hallo, I'm an Undo developer wave

Indeed, using GDB's version of tracepoints (dprintf - https://doc.ecoscentric.com/gnutools/doc/gdb/Dynamic-Printf....) is really powerful and replaying a trace with these installed is to generate "logs I wish I'd had" is exciting.

It also has potential use if:

* you'd like to diff two executions (e.g. one successful, one failing) - you probably can't just compare raw execution as there'll be uninteresting variation but comparing extended logs could be useful

* you are debugging something at a customer site - they might not let you debug directly but you could iterate on plain text logs by shipping them additional tracepoints to run on a recording

We thought this was useful enough that we implemented a tool with its own DSL for simply specifying post factor logging "probes": https://docs.undo.io/PostFailureLogging.html

[akkartik]

The big problem with debuggers in my experience is the difficulty of setup. They take a lot of code to build, and if you use a slightly different language or compiler or OS you're SoL. Or at least facing some rabbitholes of unknown depth.

The most sophisticated debuggers seem to target C or Lisp. But lately I don't use either.

I've never gotten time-travel debugging in gdb to work. And it's been out for more than 10 years at this point. The last time was 2 years ago, so I forget the details. I do love tracepoints in gdb.

I've been watching https://rr-project.org for a while. The instructions still say, "build from source".

I looked at LLDB after your previous comment. Got it installed, but I can't actually get it to set a breakpoint on Linux. The docs seem optimized for Mac.

I have no doubt that once you get something set up just right you can do great things with it. But the power to weight ratio seems totally out of whack. Part of the goal of my projects recently has been to show that you can get a bunch of features far more simply if you build on a base of logging. If the programmer is willing to modify the program, a single tool can help debug programs in a wide variety of languages. They just have to follow a common and fairly simple protocol.

The big drawback of my approaches is that they don't scale for long runs of huge codebases. That hasn't been an issue for most programs I ever want to debug. Why should I pay the complexity costs of debugging gcc or Firefox for small programs.

[jcranmer]

> I've been watching https://rr-project.org for a while. The instructions still say, "build from source".

rr should be in most distros at this point. (It's been in Debian since at least 2014).

[akkartik]

Thanks so much! I just tried it out and finally got to see time travel debugging working with my own eyes[1]. I'm glad to have it in my toolbox.

[1] Though I had to use sudo a few times, reboot, see some scary warning that it might not work reliably, read https://github.com/rr-debugger/rr/wiki/Will-rr-work-on-my-sy... and https://github.com/rr-debugger/rr/wiki/Zen.