[Veserv]

Wait, which is the hard one you wish you had time travel debugging on, the single threaded WASM context or the multithreaded native context?

The multithreaded native context is the one that is harder in principle, but should only incur ~100% overhead for any program including ones not using Bevy. Though I do not know about the general availability of these products in your field.

A single-threaded context is vastly simpler and can be done with similar overhead without platform support or ~1-10% overhead with platform support. Though I do not know is anybody has implemented efficient WASM support or if anybody with efficient multithreading implementations has ported to WASM.

Likely the only available ones are the inefficient 1,000% overhead or the hilariously bad 100,000% overhead ones like the default gdb implementation. To be fair, these implementations are much easier to write. Even ~100% overhead in the single-threaded case is more common amongst extant solutions since getting down to ~10% requires some serious optimization. Still should be perfectly adequate for development work.

[SeanAnderson]

Sounds like you know a lot more about this area than I :)

I would like an efficient way of time travelling in a single threaded context.

As you describe it, it makes sense that supporting multithreading would make the problem space much more challenging to navigate. I wasn't thinking about that, but it's clear once you point it out. I was just considering the overhead of maintaining the undo state without being able to delegate it to a separate thread.

As OP mentions, they use change detection to calculate/store deltas, but Bevy's ECS change detection isn't very performant. You still have to iterate over all components and check a component's value to learn changed state rather than being able to filter on a `Changed` archetype. It kind of makes sense, though, because adding/removing Changed components from tons of entities every tick would also be expensive. Either way, change detection feels like a sore spot when working with tons of entities in ECS. I'm not super confident there's a way around that without manually maintaining some data structures outside of the ECS paradigm, but was thinking that if I could at least run the change detection on a separate thread that it might be tolerable.

[Veserv]

If you are okay with single-threaded Linux native as a debug platform (i.e. you have a build that you reproduce bugs on) then you can probably use rr. undo.io has also been in the field for a long time. I hear they can also do multithreaded Linux native in some capacity as well. One of the people from undo frequently pops into time travel debugging threads when they appear, so they could give you more info if they drop by.

If you are on Windows, Microsoft has some form of time travel debugging, but I am pretty sure they do a instrumented emulator which is a 10-20x slowdown approach. I do not know of anything else on Windows.

The only efficient multithreaded time travel debugging I am aware of is all in the embedded field, so unlikely to be applicable. Most of the “multithreading” solutions otherwise available work by serializing your execution to a single thread, so they do not really count. Maybe there is something else out there, but not really sure.

[roca]

rr and Undo are about the same here: they support multiple threads but run all threads on a single core.

[Veserv]

Hm, thought they did more than standard replay. Do you know anybody outside of embedded that can do true multithreaded time travel debugging? I do not keep too much up to date on Linux native solutions. Most of the new ones I see are either just wrappers around rr or really hacky replay-based and should really just be wrappers around rr.

[roca]

Microsoft's TTD handles multiple threads on multiple cores, because they instrument reads. But that's high overhead.

I'd like to know more about what the embedded tools do. I see SourcePoint supports multicore x86, but the data volume looks high so I suspect they can't handle workloads that run for minutes or hours. Also it's not clear to me which of these tools can reproduce the state of memory at every point in the past.

Basically I don't think you can do low-overhead record and replay without hardware support like Intel's QuickRec project. "Capture all memory traffic" is not going to scale.